Hindsight is 2020

Ryan Benson -

Hindsight is 2020!

... ok, it's actually 20200607, but I've been waiting years to make a bad "Hindsight 2020" joke.

There's a new version of Hindsight! It's been a while, but subsequent versions should come out more regularly. There aren't many user-visible changes, but expect more changes in future versions now that the big push to Python 3 is complete.

Here's what's new:  

  • Hindsight is now in Python 3. This involved lots of code refactoring and clean-up. Things should generally run better and faster.
  • Supports Chrome versions 1 - 83.
  • New "Storage" tab/table (depending on output type) for storage artifacts that lack a timestamp (Local Storage and File System in this version; more to come).
  • More robustness all around; errors should be fewer and caught better (if not, please file an issue and let me know). I've started adding in tests and Travis CI, with more to come.
  • New versioning scheme: switch to date-based to better align with other DFIR projects (namely plaso & Timesketch). This version is 20200607.

Get It!

You can find Hindsight's code on GitHub. You can also get it by:

  • pip install pyhindsight will install the pyhindsight Python package and the hindsight.py and hindsight_gui.py scripts on your system
  • Download the compiled EXEs for Windows from the GitHub release page

Storage Artifacts

Timelines are everywhere in DFIR (for good reason), but not every artifact has a timestamp associated with it. This version of Hindsight adds a "Storage" tab to the output, where storage-related artifacts that lack timestamps can go.

New "Storage" tab in Hindsight XLSX output

Right now, the two types of records that can be in this section are File System and Local Storage. Cookies and Cache items are still in the Timeline tab, as they have reliable timestamps. I aim to add other storage types here in future versions.

Good and Bad with LevelDB

I've spent a lot of time digging into the various LevelDB databases in Chrome. Hindsight will parse out the HTML5 "File System" artifacts (not to be confused with the normal, host file system; not a good name from a DFIR point of view). The File System is another way websites can store files (similar to Local Storage, Session Storage, IndexedDB, and Cookies); see this blog post for more details. As far as I know, Hindsight is the only tool that parses this out.

Hindsight also parses out Chrome's Local Storage. Local Storage has been around for a long time. In Chrome, it started out as a bunch of SQLite databases (one for each site). It has since migrated to a single LevelDB, but the twist is existing SQLite DBs are kept too. Hindsight parses out both the SQLite files and the LevelDB database.

There are many other LevelDB artifacts in Chrome and I have a number of them parsed to some degree. I hope to add these into Hindsight in future releases.

I've switched to using a package called Plyvel for LevelDB access. The issue with it (and all other LevelDB Python packages I've found) is it works great on Linux but is hard to install on Windows.

This has been the main thing holding up this release; I've finally decided it's better to release Hindsight for Python 3 and work out the LevelDB support later than keep having that issue hold up the release. That said, if you are good with Windows and Python and can compile a wheel of Plyvel, I'd be extremely grateful.

I've left Plyvel off the pip requirements and it isn't in the compiled Windows executables. If you install Hindsight via pip (pip install pyhindsight ), you can try to install Plyvel separately (if you get it to work, please let me know!) Hindsight will still run without Plyvel, it just won't be able to access LevelDB databases. This impacts two artifact types: File System and Local Storage. File System is pure LevelDB, so if you run Hindsight without Plyvel installed, you'll see [Failed] for the count of parsed File System items. Local Storage can be a hybrid of SQLite and LevelDB, so you may see a count of parsed records, but those will only be SQLite-based, not LevelDB ones. More details about parsing actions will be in your hindsight.log file.

That's it! Take the new version for a spin and please let me know what you think!