Search text for 999999999999999999 words in no time

  • 3 Replies
  • 1241 Views
*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Search text for 999999999999999999 words in no time
« on: September 09, 2019, 02:24:48 am »
Here's one of the things I just made. Try it out. I tested it on Windows. You can search for any word in a huge amount of data really fast as if it is a 1 word search. Runs on CPU. You can swap the 200MB in the src folder. Run it in Visual Studio 2019. You can edit the words I search for in main.cpp line 199.

This is useful for when you have many many items you want to search for and you have a large amount of data to look through.

https://www.dropbox.com/s/v9vxy1bhpogppkq/FastSearch.rar?dl=0
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #1 on: September 09, 2019, 04:02:41 am »
RAM actually shrinks to ~150MB to hold ~220MB.

Binary Search would make RAM go up lots. The data stays as Huffman words.

So, it's fast+low memory.
« Last Edit: September 09, 2019, 08:18:11 am by LOCKSUIT »
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #2 on: September 09, 2019, 08:14:35 am »
There's a bug wait, it will be even faster. It's already fast.
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #3 on: September 10, 2019, 11:11:35 am »
btw if not clear: I put the search items in a tree instead of putting the data in a tree. The little tree searches the data.

update:
I made it faster:
https://www.dropbox.com/s/5lym9vl2fttrmgd/FasterFastSearch.rar?dl=0
268MB becomes 132MB and takes 12 seconds to check the full 132MB for any of 72 words (could have set it to ex. 99999 words). It takes 12 seconds no matter how many words are being searched for.
Emergent          https://openai.com/blog/

 


OpenAI Speech-to-Speech Reasoning Demo
by ivan.moony (AI News )
March 28, 2024, 01:31:53 pm
Say good-bye to GPUs...
by MikeB (AI News )
March 23, 2024, 09:23:52 am
Google Bard report
by ivan.moony (AI News )
February 14, 2024, 04:42:23 pm
Elon Musk's xAI Grok Chatbot
by MikeB (AI News )
December 11, 2023, 06:26:33 am
Nvidia Hype
by 8pla.net (AI News )
December 06, 2023, 10:04:52 pm
How will the OpenAI CEO being Fired affect ChatGPT?
by 8pla.net (AI News )
December 06, 2023, 09:54:25 pm
Independent AI sovereignties
by WriterOfMinds (AI News )
November 08, 2023, 04:51:21 am
LLaMA2 Meta's chatbot released
by 8pla.net (AI News )
October 18, 2023, 11:41:21 pm

Users Online

309 Guests, 0 Users

Most Online Today: 396. Most Online Ever: 2369 (November 21, 2020, 04:08:13 pm)

Articles