Ai Dreams Forum

Member's Experiments & Projects => General Project Discussion => Topic started by: triscabor on September 01, 2015, 08:49:47 am

Title: Non expert development of expert systems?
Post by: triscabor on September 01, 2015, 08:49:47 am
I've been racking my brain on a seemingly intractable problem: the systematic automation of typical jobs. Okay, that probably sounds a bit crazy, but here me out.

My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites and clean the data, then store it in a database.

The problem is, I'm kind of stuck -- I have all this nice data, but no real tangible way of moving forward. My initial goal consisted of two very different options:

1. Finding commonalities between discrete job tasks, and then mapping these to specific modules or processes to automate them. Then composing these into a single "job" entity. From an architectural perspective, this would be ideal. From a software engineering perspective, this is vital.
2. Recruit others in a crowd-sourcing application, either marketed as-is, or sold as another useful or entertaining destination.

These are not mutually exclusive.

I should note, these pre-empt the actual creation of an expert system -- that part would probably be a lot easier, using something like Drools for Java, or PyKe in Python. But this all hinges on having lots of domain knowledge, and knowing how to also translate it into business rules, software, etc...

Thanks for your input!
Title: Re: Non expert development of expert systems?
Post by: Korrelan on September 01, 2015, 10:44:56 am
Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

Quote
the systematic automation of typical jobs

Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...

Quote
to specific modules or processes to automate them

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol.  :D
Title: Re: Non expert development of expert systems?
Post by: ranch vermin on September 01, 2015, 11:14:58 am
yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm. :)

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.
Title: Re: Non expert development of expert systems?
Post by: triscabor on September 01, 2015, 09:28:42 pm
Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

Thanks! I wasn't sure if I was supposed to drop this in the welcome forum.

Quote
Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...


Yep, any typical employed person's job -- for simplicity let's exclude those that are physical labor or entertainment oriented, and focus more on knowledge workers, administrative jobs, etc...

In terms of granularity, It would be best if it was very very granular. Toy example:

Given an example task "write a letter", a final product like this might be:

1. Pick up writing instrument and blank piece of paper.
2. Lay paper out flat on a smooth, hard surface.
3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation.
4. Continue writing until there is no space on the current line, then move down to the next line and continue.
5. Repeat this until the writing is complete. If there is no more paper left, go to step 1.


The physical motions should have no bearing on this question -- it's just a toy example to give you an idea of granularity.

Quote

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

Nope, I already have this data, the means to continually scrape it, and the scripts to add it for use in a relational or document database.

Quote
If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol.  :D

Yep, this is what I'm after.

Title: Re: Non expert development of expert systems?
Post by: triscabor on September 01, 2015, 09:31:08 pm
yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm. :)

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.

I think my previous post might clear things up. Your above idea is pretty interesting, my only concern would be how to GET that data -- requires personal employment information that would simply be private, legally confidential, and something most people would not share (and if they did, it's hard to determine if it's trustworthy or robust enough.)
Title: Re: Non expert development of expert systems?
Post by: Korrelan on September 02, 2015, 12:09:23 am
So you need a system to break tasks/ jobs down into their base constituent parts...so a physical robot/ machine can perform them instead of a human.

If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.
Title: Re: Non expert development of expert systems?
Post by: 8pla.net on September 02, 2015, 01:52:00 am
Welcome triscabor

Quote
My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites ...

Reminds me of what I just built today, with somewhat surprising results... 

A chat bot tool kit (http://www.chatbot.tk) to scrape a major search engine to get its responses.

Object Oriented classes are best, I think, but here is some pseudo code :

Code

/* 1. Pick up writing instrument and blank piece of paper. */

      function get(pen,paper){}

/* 2. Lay paper out flat on a smooth, hard surface. */

      function put(paper){}

/* 3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation. */

/* 4. Continue writing until there is no space on the current line, then move down to the next line and continue. */


      function print(pen,text,paper){}


/* 5. Repeat this until the writing is complete. If there is no more paper left, go to step 1. */


      var pen,text,paper;

      for loop {

      function get(pen,paper){}

      function put(paper){}

      function print(pen,text,paper){}

      }  loop




I hope this helps you think about your knowledge base and inference engine design.


_________________________________________________________________________
Title: Re: Non expert development of expert systems?
Post by: Zero on September 02, 2015, 08:25:49 am
Hi triscabor,

Quote
I have all this nice data

Can we see a sample of what you've got so far please?
Title: Re: Non expert development of expert systems?
Post by: ranch vermin on September 02, 2015, 09:04:09 am
Taking one of our common day activities and putting it into an exact step by step is a little daunting a problem,  i recommend taking a step back, have a think and try to find some unified way to solve a fair degree of problems with one method, (what im doing.) because the job is massive, handling seemingly simple day to day tasks - and could get needlessly complex.
Title: Re: Non expert development of expert systems?
Post by: Korrelan on September 02, 2015, 11:34:32 am
Quote
If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code
Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.
Title: Re: Non expert development of expert systems?
Post by: ranch vermin on September 02, 2015, 01:30:27 pm
the hierarchy of functions seems like a good idea.  Maybe if you go to some level of low level, without actually making it all the way to moving the fingers/arm etc, could be a good way to start.
Title: Re: Non expert development of expert systems?
Post by: 8pla.net on September 02, 2015, 04:11:39 pm
I ported an expert system C language source code to compile on the Linux operating system using g++ in a Bash Terminal.  For all you Windows users out there, g++ is a Linux C++ compiler.

Would you like me to post my working expert system C language source code listing as a foundation for study?   If not, maybe eventually, so I don't break the flow of this discussion... I'll start a new separate topic on programming an Expert System in the C Language on Linux.

Reference:  http://www.drdobbs.com/rule-based-programming-in-c/199702471?pgno=4 (http://www.drdobbs.com/rule-based-programming-in-c/199702471?pgno=4)
Title: Re: Non expert development of expert systems?
Post by: triscabor on September 03, 2015, 08:35:43 am
Hi triscabor,

Quote
I have all this nice data

Can we see a sample of what you've got so far please?

The project lives heres: https://github.com/christabor/jobScraper

I don't commit data files to source, but you can run it yourself if you'd like. I've uploaded example json output here though

Industry (category) and job titles:
https://gist.github.com/christabor/217cb3fdd6c4fc96fe35

Single job title
https://gist.github.com/christabor/553fc8cd320c1359099e

Title: Re: Non expert development of expert systems?
Post by: triscabor on September 03, 2015, 08:49:28 am
Quote
If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code
Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.


I've had the basic notion of decomposition in my head, but just the act of practicing and spelling it out seems to help. One thing that just recently occurred to me is the possibility of using Natural Language Processing to "assemble" the decomposed version (I believe the notion of realizing a concept into concrete code and data models is referred to as reification). In fact, I actually did a very naive toy version of exactly this for a separate project of mine: https://github.com/christabor/MoAL/blob/master/MOAL/software_engineering/reification/reification.py

The idea is to generate a parse tree and then using the different part-of-speech tokens, output actual code from each one. So, in this case, verbs map to functions and nouns map to classes. Unfortunately it's not very robust so the wrong nouns often map to a class, because it does not determine the subject correctly -- but this could be overcome with more robust code.

With this in mind, I could imagine taking all the "tasks" for which I have raw, structured data, analyzing the parts of speech for each discrete task, and then generating pseudocode for all tasks for all jobs. This could be a first stepping stone. The nice thing about this is that duplicate tasks across seemingly disparate jobs could become unified and make it easier to "divide and conquer" the problem space.

I think just talking about the problem is having a positive effect on my thinking!
Title: Re: Non expert development of expert systems?
Post by: ranch vermin on September 03, 2015, 11:22:16 am
Your english wont be complete enough without ALOT of guesswork on behalf of the NLP algorythm, because when we explain things 99% of the activity is omitted from the sentence entirely.

This is an AGI attempt, and its an unsolved problem to date.

Again ill suggest what I suggested before - go to some level of specificy, but dont go any further at first,  itll be alot easier for anyone.
Title: Re: Non expert development of expert systems?
Post by: 8pla.net on September 03, 2015, 05:17:52 pm
Here is some more sample data:

Code
{"data":[{"jobs":[
    {"whitecollar":[
        {"id":0,"title":"Office Assistant"},
        {"id":1,"title":"Office Manager"},
        {"id":2,"title":"Executive"}
    ]},
    {"bluecollar":[
        {"id":0,"title":"Fast Food Worker"}
        {"id":1,"title":"Gas Station Attendant"}
    ]}]
},{"outsourced":[
        {"id":0,"title":"Office Assistant"},
        {"id":1,"title":"Office Manager"},
        {"id":2,"title":"Executive"}
    ]
}]}

 ;)
Title: Re: Non expert development of expert systems?
Post by: Korrelan on September 04, 2015, 01:10:29 pm
Using common coding methods this is going to be an extremely difficult task, especially given the nature of the input. I think your best bet is to build a system that would improve over time with a lot of human guidance and effort. eg...

Possible method...

Start by either obtaining or parsing a dictionary listing words and their related grammar designation, verb, noun, etc.

Then parse as many sample sentences as possible creating sentence templates replacing the actual words with tags. Remove punctuation and replace ',' with 'and', etc.

so... Communicating effectively in writing as appropriate for the needs of the audience.

becomes... <verb1> <adverb1> in <noun1> as <adjective1> for the <adverb2> of the <noun2>.

These could possibly be split down even further.

Once you have the various templates assign know actions/ functions to the groups of templates that 'mean' similar things. so.. the above example would perhaps create...

required_skill(<verb1>,<adverb1>)
required_action(<noun1>)

Use a percentage scoring routine (blackboard) to match each parsed input sentence with your list of templates to find the best match on a word type/ word position basis.

I would also run a small 'current memory' stack that would bias the matching (a chat bot technique). So if a sentence says... regarding the skill.. 'skill' would be added to the sentence template as a preamble, as well as kept in the current memory / topic store whilst parsing the following input sentences.

Although the coding required is simple, I think the amount of time and effort required to build the parser would be detrimental to your sanity. :)
Title: Re: Non expert development of expert systems?
Post by: triscabor on September 09, 2015, 08:15:06 pm
Thanks all for the feedback. I took a stab at handwriting my POC function decomposition for a few tasks, and it was definitely challenging to create something of substance even using my own brain. I also spent some time playing with different NLP tools, notably NLTK and http://www.clips.ua.ac.be/pages/pattern-en, (http://www.clips.ua.ac.be/pages/pattern-en,) and it yielded some very interesting results, but I find this is going into a rabbit hole the more I dig into it (specifically, determining relationships between words to parse as "method names" -- I have made some progress but it's still way too infantile.)

I think I'll take a break from this front and try a few other ideas and projects until I can muster the energy to come back to it.