Using boto3, moto and freezegun in a Python test

Recently I worked on a Python script to monitor and delete objects inside an S3 bucket.
I published an excerpt on GitHub: python-projects/delete-s3-objects

The code in the repo mostly consist of the cleanup() function. The function use boto3 to connect to AWS, pull a list of all the objects contained in a specific bucket and then delete all the objects older than n days.
I have included a few examples of creating a boto3.client which is what the function is expecting as the first argument. The other arguments are used to build the path to the directory inside the S3 bucket where the files are located. This path in AWS terms is called a Prefix.

As the number of the objects in the bucket can be larger than 1000, which is the limit for a single GET in the GET Bucket (List Objects) v2, I used a paginator to pull the entire list. The objects removal follow the same principle and process batches of 1000 objects.

Now this was all good fun but the really interesting part was creating a proper unittest.

After some searching I found moto, the “Mock AWS Services” library. It is brilliant!
Using this library the test will mock access to the S3 bucket and create several objects in the bucket. You can leave the dummy AWS credentials in the script as they won’t be needed.

At this point I wanted to create multiple objects in the S3 mocked environment with different timestamps, but unfortunately I discovered that this is not possible. Once an object is created in S3 the date of creation metadata cannot be easily altered, see here for reference.

Cue another awesome library called freezegun. The test use freeze_time to mock the date/time and create S3 objects with different timestamps, so that we can safely experiment with the logic of the cleanup() function (‘leave objects older than n days, delete everything else within the prefix‘).

$ python 
mock-root-prefix/mock-sub-prefix/test_object_01 2019-08-29 00:00:00+00:00
mock-root-prefix/mock-sub-prefix/test_object_02 2019-08-28 00:00:00+00:00
mock-root-prefix/mock-sub-prefix/test_object_03 2019-08-27 00:00:00+00:00
mock-root-prefix/mock-sub-prefix/test_object_04 2019-08-26 00:00:00+00:00
mock-root-prefix/mock-sub-prefix/test_object_05 2019-08-25 00:00:00+00:00
mock-root-prefix/mock-sub-prefix/test_object_06 2019-08-24 00:00:00+00:00
<class 'botocore.client.S3'>
Cleanup S3 backups
Working in the bucket:         my-mock-bucket
The prefix is:                 mock-root-prefix/mock-sub-prefix/
The threshold (n. days) is:    4
Total number of files in the bucket:     7
Number of files to be deleted:           3
Deleting the files from the bucket ...
Deleted:        3
Left to delete: 0
Ran 1 test in 0.798s


Upgrading to Ubuntu 18.04

I really like Ubuntu and I use it in different flavours on most of my computers at home. I tend to stick to the LTS and upgrade when the xx.04.1 version is out (in other words, when it prompts me to do so).

I know a fresh installation is usually the best but I also like to just do the upgrade and so far it worked perfectly for me. Except in one case.

I am going to put here my fix, hope it can help others.

from 16.04 to 18.04

The upgrade worked without any problem on all my machines except this one laptop. The laptop was previously upgraded from a 14 LTS and this could be the reason.

To prepare for upgrade I first ran

apt-get update && apt-get upgrade

Waited until it finished and successfully updated the 16.04 LTS, I didn’t need to run a dist-upgrade as there wasn’t any package held back.

I started the release upgrade from the terminal and everything seemed to go well until it failed:

# do-release-upgrade 
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [819 B] 
Get:2 Upgrade tool [1,258 kB] 
Fetched 1,258 kB in 0s (0 B/s) 
authenticate 'bionic.tar.gz' against 'bionic.tar.gz.gpg' 
extracting 'bionic.tar.gz'
Inhibiting until Ctrl+C is pressed...
Fetched 0 B in 0s (0 B/s)
Processing triggers for libc-bin (2.27-3ubuntu1) ...
(Reading database ... 228417 files and directories currently installed.)
Removing systemd-shim (9-1bzr4ubuntu1) ...
Removing 'diversion of /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service to /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd by systemd-shim'
dpkg-divert: error: rename involves overwriting '/usr/share/dbus-1/system-services/org.freedesktop.systemd1.service' with
different file '/usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.systemd', not allowed
dpkg: error processing package systemd-shim (--remove):
installed systemd-shim package post-removal script subprocess returned error exit status 2
Errors were encountered while processing:
Upgrade complete

The upgrade has completed but there were errors during the upgrade

To continue please press [ENTER

Popped into a console and tried to fix it manually and force remove systemd-shim but it didn’t work. After a quick search I was able to resolve the problem renaming the file and forcing the pending installation to be completed

# mv /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service{,bak}

# apt-get -f install

That worked but just to stay on the safe side I then ran my usual upgrade routine which consists of:

apt-get update && apt-get upgrade
apt-get dist-upgrade
apt-get autoremove && apt-get clean

Followed by another do-release-upgrade which happily showed that there was nothing else left. A quick reboot and we should be good, right? Wrong.

where is the dock?

The system booted and I was greeted by the new beautiful log in screen. Enter the password and… where is the dock?

Pressing the “windows” key I was able to see the default gnome dock and all my applications as Favourites, but the new shiny Ubuntu dock was nowhere to be found.

Even opening the settings showed a completely empty ‘dock’ section.

I tried installing gnome-tweaks but still no joy. In the end I figured out that something (an extension probably) was missing and a quick apt-cache search confirmed my suspicion.

# apt-get install gnome-shell-extension-appindicator \
gnome-shell-extension-ubuntu-dock \
gnome-shell-extension-system-monitor \
gnome-shell-extension-top-icons-plus \

After installing a few extensions I had the dock back and the configuration showing in the Settings. Not sure if all of these are needed, probably the ubuntu-dock alone would do.

I am on TPTM podcast

Yes, I am on Talk Python To Me podcast Episode #174: Coming into Python from another Industry (part 2).

Now I have a very good excuse to brush up my github and start adding all the stuff I have been working on.

My first script

I have finished my first real attempt at Python.

Nothing too complicated, just a simple script:

  • Python 3 (of course!)
  • minimal modules for max portability
  • pull information from Xymon and parse it
  • pull RAID and disk information using omreport and parse it to obtain a list of disks failed/in predictive failure, serial numbers, etc.
  • generate a report and print a template
  • follow the disk rebuild when the option -p is used, wrap the screen refresh in curses and wait for ‘q’ to be pressed then exit

I am sure it can be improved and made a lot better, as well as I am sure I did some horrible mistake somewhere. But it’s a start.

Find it on GitHub:

Now that this project is finished I can focus on the #100DaysOfCode in Python.

TIL Python3 functions all() and str()

All this time, the solution to my problems was just in front of my eyes.

While I am procrastinating getting ready for the 100DaysOfCode challenge, I am working on a project to rewrite a bash script in Python3.

Today I solved two problems in one go. I am sure the experienced coder would have done it in a minute and with time to spare, but hey that’s how we learn.

My first problem was to nicely transform a list of strings in a way that would make it easier to do multiple regex search on it.

The list looks like this:

ID : 0:0:2
Status : Non-Critical
Name : Physical Disk 0:0:2
State : Online
Failure Predicted : Yes
Progress : Not Applicable
Bus Protocol : SAS
Media : HDD

It is the result of a command (omreport storage pdisk controller=0) that gathers information about the disks status.

After a closer look I discovered that each item of the list is a byte object (b'ID : 0:0:2') and needs to be transformed to a string type. Also, I wanted to make a nice block for each disk.

This did the trick:

for i in omreport:
    disk_string += str(i, 'utf-8')
blocks = disk_string.split("\n\n")

Note the ‘utf-8’ encoding. More here.

Now blocks is a list of multi-line strings (real strings!) that can be processed with re.findall:

found1 = re.findall(r'^ID\s+\:\s(.*)\n', block, re.MULTILINE)
found2 = re.findall(r'^Status\s+\:\s(.*)\n', block, re.MULTILINE)
found3 = re.findall(r'^State\s+\:\s(\w+)\n', block, re.MULTILINE)

From here we can build a tuple formed by all the information needed and work with that.

found = (found1, found2, found3)

But the last bit is: how can we make sure that the tuple is not empty? How can we throw away something like this: ([], [], []).

And here comes all() to the rescue. More here.

This, again, did the trick:

if all(found):

100DaysOfCode checklist

I am about to start my 100DaysOfCode challenge in Python.

I am assuming everyone is familiar with the concept, but if you want to know more here’s some reading/listening material:

I will be following the course “#100DaysOfCode in Python” by Michael Kennedy, Bob Belderbos and Julian Sequeira.

Try is my pre-flight checklist:

  • ✓ setup a dedicated python3 environment on my DigitalOcean development droplet
  • clone the course GitHub repository
  • ✓ tune on an online radio to help focus (either SomFM: Groove Salad or Radio Swiss Classic)
  • ✓ undust my website (last post is dated 2015!) and my twitter account
  • code

(mostly a reminder to myself)

Ugrade from Wheezy to Jessie

The time has come to upgrade my BanaPi (running Bananian) from Debian 7 (Wheezy) to Debian 8 (Jessie).
I am thankful for the amazing documentation. It’s going to be fun.

Other small projects I am running at the moment: solve a kernel panic on a Gentoo box, set up services on various VMs that I am running for experiment/fun/learning.

Study for my certifications and working on different Linux projects, this is a good way to spend a Satuday isn’t it?
But then I deserve a nice dinner out with my gf.


It’s quite fun if I think to it: I am pretty sure one of the old posts on the old version of my website was about me having fun installing Gentoo.

And now, here I am again: having fun with Gentoo.
Just created a VM and going to run again through the amazing documentation and the Gentoo Wiki.

For science!

Speaking of the old website, I need to find time to either import it into WordPress or just convert and link to the static version, which may be a lot easier…


Podcast fun

This is a bit new to me.

In the past I rarely had time to listen to anything going to work. And to be fair I had nothing, as in a device or a player, that would make it easier to play podcasts.

Now I am quite enjoying to listen to podcasts on my way to work. Most of them are related to Linux, this is my (little) list so far in an absolutely random order:

Any suggestion is more than welcome.

6 Sept 2015, added a link to Linux Luddites