New Hindsight Release: Better LevelDB parsing, New Web UI View, & More!

It's been a while, but a new Hindsight release is here! This new version (2021.01.16) brings exciting new features: improved LevelDB parsing (including deleted!), viewing Hindsight results in the web UI, and more!

Improved LevelDB Parsing

LevelDB has been used in Chrome for years... and for years I've had difficulties parsing it. The Python support for LevelDB hasn't been great; all the Python packages required you to have LevelDB installed on the system already and they acted like a shim to it. This worked great on Linux systems, as LevelDB was (relatively) easy to install, but proved a challenge on Windows systems.  

Then Alex Caithness from CCL Forensics came out with a couple of fantastic blog posts (and code!) exploring Chrome's IndexedDB. IndexedDB in Chrome is complicated in its own right, but it also uses LevelDB for data storage. In Alex's exploration of IndexedDB, he created a pure Python parser for LevelDB! This code (which he released as open source), makes reading LevelDB in Python a lot easier. I've switched Hindsight over to using ccl_chrome_indexeddb for reading LevelDB and removed the old code and dependencies, which means Hindsight should parse LevelDB records now out of the box on all platforms!

Right now, FileSystem and LocalStorage records are the only LevelDB-backed artifacts that Hindsight parses, but I'll be adding more in the coming months. Both these record types appear in the "Storage" tab. Thanks to Alex's code, I was able to add a two new columns (Sequence and State), both about the LevelDB internals; I'll expand on them in a later post. The File System records got a few more additional columns, thanks to suggestions from Chad Tilbury, that help you see what files still exist on disk and a bit about them (size and type).

Bonus: Deleted Records!

One of the things that excited me initially when I was digging into LevelDB is that the format lends itself to keeping deleted records around for a while. I've been using a golang program called ldbdump to explore deleted records, and you can find a lot of them! Another great thing about the switch to using the CCL Forensics' code in Hindsight is that since it parses deleted records, Hindsight now can too! More to come on this in a later post.

Viewing SQLite Results in Hindsight's Web UI

Since Hindsight's beginning, it has been a parsing tool; you would have to view that parsed output somewhere else (an XLSX file in Excel, or maybe a JSONL file loaded into Timesketch). Thanks to Ryne Everett, you can now view parsed records in Hindsight too! He's added the ability to view Hindsight's SQLite output in the Hindsight web UI. It uses his sqlite-view project, which is based on sqlite-viewer, to add a SQL-like view and querying interface to Hindsight.

After running Hindsight's web UI and processing some browser history files, there's a new button (View SQLite DB in Browser). After clicking that, a view like the above screenshot will appear. You can select which table to view by clicking the table name at the top, and you can do SQLite queries as if you were in an external SQLite viewer.

It does require a separate install step, as we didn't want to bundle all the supporting Javascript code in the Hindsight repo. If you don't have the necessary Javascript code installed, you just won't be able to use the new functionality (the button will be grayed out); everything else in Hindsight should continue to work as normal. I've included these supporting files in the compiled EXE version, so this feature is enabled in it.

Parsing "Media History" Artifacts

Chrome added a new "Media History" database in version 86, and this version of Hindsight adds support for parsing it. See this blog post for more info on this new artifact.

Update Minimum Python version to 3.8

The switch to using the CCL Forensics LevelDB parsing code necessitated moving Hindsight to use Python 3.8, rather than 3.7. I hope this isn't too big an issue for anyone, as 3.7 has moved to security-fixes only and 3.8 (and 3.9) have performance improvements as well.

Get Hindsight

You can get Hindsight, view the code, and see the full change log on GitHub. Both the command line and web UI versions of this release are available as:

• compiled exes attached to the GitHub release or in the dist/ folder
• .py versions are available by pip install pyhindsight or downloading/cloning the GitHub repo.

NOTE: Windows Defender has been flagging the EXEs as malware, presumably because they were packaged with PyInstaller. The Python script versions are not being flagged. If you'd like to build the EXEs from the Python code yourself, all I did was: pyinstaller --distpath .\dist .\spec\hindsight.spec from the root of the repo.