Search text for 999999999999999999 words in no time

  • 3 Replies
  • 1511 Views
*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Search text for 999999999999999999 words in no time
« on: September 09, 2019, 02:24:48 am »
Here's one of the things I just made. Try it out. I tested it on Windows. You can search for any word in a huge amount of data really fast as if it is a 1 word search. Runs on CPU. You can swap the 200MB in the src folder. Run it in Visual Studio 2019. You can edit the words I search for in main.cpp line 199.

This is useful for when you have many many items you want to search for and you have a large amount of data to look through.

https://www.dropbox.com/s/v9vxy1bhpogppkq/FastSearch.rar?dl=0
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #1 on: September 09, 2019, 04:02:41 am »
RAM actually shrinks to ~150MB to hold ~220MB.

Binary Search would make RAM go up lots. The data stays as Huffman words.

So, it's fast+low memory.
« Last Edit: September 09, 2019, 08:18:11 am by LOCKSUIT »
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #2 on: September 09, 2019, 08:14:35 am »
There's a bug wait, it will be even faster. It's already fast.
Emergent          https://openai.com/blog/

*

LOCKSUIT

  • Emerged from nothing
  • Trusty Member
  • *******************
  • Prometheus
  • *
  • 4659
  • First it wiggles, then it is rewarded.
    • Main Project Thread
Re: Search text for 999999999999999999 words in no time
« Reply #3 on: September 10, 2019, 11:11:35 am »
btw if not clear: I put the search items in a tree instead of putting the data in a tree. The little tree searches the data.

update:
I made it faster:
https://www.dropbox.com/s/5lym9vl2fttrmgd/FasterFastSearch.rar?dl=0
268MB becomes 132MB and takes 12 seconds to check the full 132MB for any of 72 words (could have set it to ex. 99999 words). It takes 12 seconds no matter how many words are being searched for.
Emergent          https://openai.com/blog/

 


Requirements for functional equivalence to conscious processing?
by DaltonG (General AI Discussion)
November 19, 2024, 11:56:05 am
Will LLMs ever learn what is ... is?
by HS (Future of AI)
November 10, 2024, 06:28:10 pm
Who's the AI?
by frankinstien (Future of AI)
November 04, 2024, 05:45:05 am
Project Acuitas
by WriterOfMinds (General Project Discussion)
October 27, 2024, 09:17:10 pm
Ai improving AI
by infurl (AI Programming)
October 19, 2024, 03:43:29 am
Atronach's Eye
by WriterOfMinds (Home Made Robots)
October 13, 2024, 09:52:42 pm
Running local AI models
by spydaz (AI Programming)
October 07, 2024, 09:00:53 am
Hi IM BAA---AAACK!!
by MagnusWootton (Home Made Robots)
September 16, 2024, 09:49:10 pm
LLaMA2 Meta's chatbot released
by spydaz (AI News )
August 24, 2024, 02:58:36 pm
ollama and llama3
by spydaz (AI News )
August 24, 2024, 02:55:13 pm
AI controlled F-16, for real!
by frankinstien (AI News )
June 15, 2024, 05:40:28 am
Open AI GPT-4o - audio, vision, text combined reasoning
by MikeB (AI News )
May 14, 2024, 05:46:48 am
OpenAI Speech-to-Speech Reasoning Demo
by MikeB (AI News )
March 31, 2024, 01:00:53 pm
Say good-bye to GPUs...
by MikeB (AI News )
March 23, 2024, 09:23:52 am
Google Bard report
by ivan.moony (AI News )
February 14, 2024, 04:42:23 pm
Elon Musk's xAI Grok Chatbot
by MikeB (AI News )
December 11, 2023, 06:26:33 am

Users Online

502 Guests, 0 Users

Most Online Today: 597. Most Online Ever: 2369 (November 21, 2020, 04:08:13 pm)

Articles