[frankinstien]

Currently a H100 at its lowest price is about $35,000! The A100s new are about $6200, and used I've seen as low as $4500, but the A100 can't do fp8. However the RTX 4090 does support fp8, but its version is 8.9 and the H100 version is 9.0. So the cuda libraries only support fp8 for the H100, and Nvidia seems to be crawly at opening up fp8 for version 8.9. OK, but there are some other open source libs that do support fp8 for the RTX4090 and you can get by. So hears the problem the Nvidia specs for the RTX 4090 states that is supports fp8 and it can do a petaflop+ in performance. A white paper was written up using a RTX 4090 that was OCed and the best they could get with fp8 is 660 TFLOPs. This is frustrating since I almost bought a RTX 4090 because Nvidia states it can do 1 petaflop+, but apparently that is a lie...

The fp16 using the Tensor cores of the RTX 30 series, specifically for RTX 3090 ti is 320 TFLOPs. Where latest prices for RTX 3090 ti used is $720 to $1200. Considering two RTX 3090 ti at the low end would be $1440 where you get at least 48GB of memory between the two, as an enthusiast, it might be a better route.

The RTX 4090s go as low as $1699 to $2800, if you go the the Nvidia shop links.

[8pla.net]

That's a great idea! It would certainly be beneficial to get two RTX 3090 ti cards at the lower end of the price range. You'd get double the memory, and if you have the right setup to support two cards, this could bring a significant boost in performance. Be sure to do your research to ensure it makes sense for your particular needs.