The "JPEG" compression artifacts are likely due to the onboard wavelet compression, necessary because of limits of onboard memory and downlink bandwidth (remember that MESSENGER does not have a HGA because its pointing is so tightly constrained by the thermal environment; instead it has a steerable phase-array thingamabob that they call the "MGA"). From http://img.pds.nasa.gov/documentation/MDIS_CDR_RDRSIS.PDF:
2.1.3 MDIS Data Compression
The MESSENGER mission requires compression to meet its imaging objectives within the available downlink. Figure 2-5 summarizes the compression options available to MDIS at the instrument level using the DPU and the spacecraft main processor (MP). At the focal plane, 2×2 binning is available on-chip to reduce the 1024×1024 images to 512×512 format. In the DPU, 12-bit data number (DN) levels can be companded to 8 bits, and data can be compressed losslessly. The strategy for DPU image compression is to acquire all monochrome data in 8-bit mode, and color data in 12-bit mode, and to compress all data losslessly to conserve recorder space. After data are written to the recorder, they can be uncompressed and recompressed by the MP more aggressively using any of several options: additional pixel-binning, subframing, and lossy compression using an integer wavelet transform. The nominal strategy for MP compression is that all data except flyby color imaging will be wavelet compressed, typically 8:1 for monochrome data and to a lower ratio (≤ 4:1) for orbital color data. Color imaging but not monochrome imaging may be further pixel-binned. For the special case of optical navigation images, there is a “jailbar” option that saves selected lines of an image at a fixed interval for optical navigation images of Mercury during flyby approaches.
Compression performance was extensively modeled prior to launch. The 12-to-8 bit look-up tables have been designed to preferentially retain information at low, medium, or high 12-bit DN values, for a nominal detector bias or for one that has decreased with time (Figure 2-6). Compression ratios to be used for flight have been based on a study of the magnitude and spatial coherence of compression artifacts using NEAR images. For expected loading of the main processor, simulations have shown that the MP can compress the equivalent of 82 full 1024×1024 images per day (or 330 512×512 images per day). The actual number of images has also been simulated, based on orbital trajectory simulations and the imaging plan described below. The MP image compression capabilities are consistent with the mission-average number of images per day. However, on days when lighting is favorable for global mapping, a peak of ~260 images per day may be expected, requiring on-chip binning of most of the data on peak days. The full implications for average imaging resolution are still being assessed.