Author Topic: Non expert development of expert systems?  (Read 2675 times)

triscabor

  • *
  • Roomba
  • Posts: 6
Non expert development of expert systems?
« on: September 01, 2015, 07:37:47 AM »
I've been racking my brain on a seemingly intractable problem: the systematic automation of typical jobs. Okay, that probably sounds a bit crazy, but here me out.

My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites and clean the data, then store it in a database.

The problem is, I'm kind of stuck -- I have all this nice data, but no real tangible way of moving forward. My initial goal consisted of two very different options:

1. Finding commonalities between discrete job tasks, and then mapping these to specific modules or processes to automate them. Then composing these into a single "job" entity. From an architectural perspective, this would be ideal. From a software engineering perspective, this is vital.
2. Recruit others in a crowd-sourcing application, either marketed as-is, or sold as another useful or entertaining destination.

These are not mutually exclusive.

I should note, these pre-empt the actual creation of an expert system -- that part would probably be a lot easier, using something like Drools for Java, or PyKe in Python. But this all hinges on having lots of domain knowledge, and knowing how to also translate it into business rules, software, etc...

Thanks for your input!

korrelan

  • *
  • Replicant
  • ********
  • Posts: 568
  • Look into my eyes! WOAH!
    • Main YouTube
Re: Non expert development of expert systems?
« Reply #1 on: September 01, 2015, 09:32:56 AM »
Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

Quote
the systematic automation of typical jobs

Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...

Quote
to specific modules or processes to automate them

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol.  :D
It thunk... therefore it is!

ranch vermin

  • Not much time left.
  • *******
  • Starship Trooper
  • Posts: 366
  • Not much time.
Re: Non expert development of expert systems?
« Reply #2 on: September 01, 2015, 10:02:58 AM »
yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm. :)

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.
« Last Edit: September 01, 2015, 12:20:58 PM by ranch vermin »
A bit from here, a bit from there, and bring it together and see the whole picture.

triscabor

  • *
  • Roomba
  • Posts: 6
Re: Non expert development of expert systems?
« Reply #3 on: September 01, 2015, 08:16:42 PM »
Hi Triscabor

I believe this is your first post, so welcome, also... I'm only on my third coffee so I'm not firing on all cylinders yet...

Thanks! I wasn't sure if I was supposed to drop this in the welcome forum.

Quote
Firstly by 'job' I presume you mean employment? Are you wanting to automate the actual job (by machine) or generate a list of physical/ mental requirements to do a specific job. Also this can mean any job from a road sweeper to neuro scientist depending on which circles the person replying to your query moves in. 

At what level/ resolution are you storing your data ie: 1 Turn up,  2 Do job, 3 Go home, lol, any job can be broken down into 'sub' jobs... and so on...


Yep, any typical employed person's job -- for simplicity let's exclude those that are physical labor or entertainment oriented, and focus more on knowledge workers, administrative jobs, etc...

In terms of granularity, It would be best if it was very very granular. Toy example:

Given an example task "write a letter", a final product like this might be:

1. Pick up writing instrument and blank piece of paper.
2. Lay paper out flat on a smooth, hard surface.
3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation.
4. Continue writing until there is no space on the current line, then move down to the next line and continue.
5. Repeat this until the writing is complete. If there is no more paper left, go to step 1.


The physical motions should have no bearing on this question -- it's just a toy example to give you an idea of granularity.

Quote

If your just wanting to enter a job title into a system and retrieve a list requirements a person would need to do said job, then a simple relational database should do the job.

Nope, I already have this data, the means to continually scrape it, and the scripts to add it for use in a relational or document database.

Quote
If your trying to create a system that can actually perform the jobs/ tasks, well... that's a slightly harder problem lol.  :D

Yep, this is what I'm after.


triscabor

  • *
  • Roomba
  • Posts: 6
Re: Non expert development of expert systems?
« Reply #4 on: September 01, 2015, 08:19:08 PM »
yeh I didnt quite understand either.

You want to automate a job allocator computer???   

If so.

a) make a 1024 bit descriptor for what a job is.  which could include information about the person that went for it.

b) then put them into categories automatically, by designating some kind of boundaries and then dragging the jobs into each other due to similarity, as they pop up to the system.

c) have a method that stands for if the person got the job or not, or if he did any good at it,  or got fired.

d) take the person data, then graft the perfect person for the job, with a genetic algorythm. :)

Maybe not that exactly,  but machine learning does stuff like that, im not sure what you are after to cater your functionality.

I think my previous post might clear things up. Your above idea is pretty interesting, my only concern would be how to GET that data -- requires personal employment information that would simply be private, legally confidential, and something most people would not share (and if they did, it's hard to determine if it's trustworthy or robust enough.)

korrelan

  • *
  • Replicant
  • ********
  • Posts: 568
  • Look into my eyes! WOAH!
    • Main YouTube
Re: Non expert development of expert systems?
« Reply #5 on: September 01, 2015, 10:57:23 PM »
So you need a system to break tasks/ jobs down into their base constituent parts...so a physical robot/ machine can perform them instead of a human.

If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.
It thunk... therefore it is!

8pla.net

  • *
  • Terminator
  • *********
  • Posts: 752
    • 8pla.net
Re: Non expert development of expert systems?
« Reply #6 on: September 02, 2015, 12:40:00 AM »
Welcome triscabor

Quote
My first goal was to aggregate a ton of supporting data to kick things off. I've done that by writing a few tools and applications to scrape numerous job sites ...

Reminds me of what I just built today, with somewhat surprising results... 

A chat bot tool kit to scrape a major search engine to get its responses.

Object Oriented classes are best, I think, but here is some pseudo code :

Code: [Select]

/* 1. Pick up writing instrument and blank piece of paper. */

      function get(pen,paper){}

/* 2. Lay paper out flat on a smooth, hard surface. */

      function put(paper){}

/* 3. With writing instrument, draw in a left to right fashion (assuming english or latin language), individual characters that form each word, followed by appropriate punctuation. */

/* 4. Continue writing until there is no space on the current line, then move down to the next line and continue. */


      function print(pen,text,paper){}


/* 5. Repeat this until the writing is complete. If there is no more paper left, go to step 1. */


      var pen,text,paper;

      for loop {

      function get(pen,paper){}

      function put(paper){}

      function print(pen,text,paper){}

      }  loop




I hope this helps you think about your knowledge base and inference engine design.


_________________________________________________________________________
« Last Edit: September 02, 2015, 01:58:02 AM by 8pla.net »
My Very Enormous Monster Just Stopped Using Nine

Zero

  • *
  • Autobot
  • ******
  • Posts: 204
  • Fictional character
    • AGI DEV LOG
Re: Non expert development of expert systems?
« Reply #7 on: September 02, 2015, 07:13:49 AM »
Hi triscabor,

Quote
I have all this nice data

Can we see a sample of what you've got so far please?
PRO|STAMINA [ programming game ]
AGI DEV LOG [ In Goto We Trust ]

ranch vermin

  • Not much time left.
  • *******
  • Starship Trooper
  • Posts: 366
  • Not much time.
Re: Non expert development of expert systems?
« Reply #8 on: September 02, 2015, 07:52:09 AM »
Taking one of our common day activities and putting it into an exact step by step is a little daunting a problem,  i recommend taking a step back, have a think and try to find some unified way to solve a fair degree of problems with one method, (what im doing.) because the job is massive, handling seemingly simple day to day tasks - and could get needlessly complex.
A bit from here, a bit from there, and bring it together and see the whole picture.

korrelan

  • *
  • Replicant
  • ********
  • Posts: 568
  • Look into my eyes! WOAH!
    • Main YouTube
Re: Non expert development of expert systems?
« Reply #9 on: September 02, 2015, 10:22:32 AM »
Quote
If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code: [Select]
Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.
« Last Edit: September 02, 2015, 11:39:58 AM by korrelan »
It thunk... therefore it is!

ranch vermin

  • Not much time left.
  • *******
  • Starship Trooper
  • Posts: 366
  • Not much time.
Re: Non expert development of expert systems?
« Reply #10 on: September 02, 2015, 12:18:27 PM »
the hierarchy of functions seems like a good idea.  Maybe if you go to some level of low level, without actually making it all the way to moving the fingers/arm etc, could be a good way to start.
A bit from here, a bit from there, and bring it together and see the whole picture.

8pla.net

  • *
  • Terminator
  • *********
  • Posts: 752
    • 8pla.net
Re: Non expert development of expert systems?
« Reply #11 on: September 02, 2015, 02:59:39 PM »
I ported an expert system C language source code to compile on the Linux operating system using g++ in a Bash Terminal.  For all you Windows users out there, g++ is a Linux C++ compiler.

Would you like me to post my working expert system C language source code listing as a foundation for study?   If not, maybe eventually, so I don't break the flow of this discussion... I'll start a new separate topic on programming an Expert System in the C Language on Linux.

Reference:  http://www.drdobbs.com/rule-based-programming-in-c/199702471?pgno=4
My Very Enormous Monster Just Stopped Using Nine

triscabor

  • *
  • Roomba
  • Posts: 6
Re: Non expert development of expert systems?
« Reply #12 on: September 03, 2015, 07:23:43 AM »
Hi triscabor,

Quote
I have all this nice data

Can we see a sample of what you've got so far please?

The project lives heres: https://github.com/christabor/jobScraper

I don't commit data files to source, but you can run it yourself if you'd like. I've uploaded example json output here though

Industry (category) and job titles:
https://gist.github.com/christabor/217cb3fdd6c4fc96fe35

Single job title
https://gist.github.com/christabor/553fc8cd320c1359099e


triscabor

  • *
  • Roomba
  • Posts: 6
Re: Non expert development of expert systems?
« Reply #13 on: September 03, 2015, 07:37:28 AM »
Quote
If this is your requirement then I'm afraid there is no solution generally available to the public, many institutions/ corporations are working on the problem and it comes under the banner of AGI.

Perhaps my last comment wan't that constructive lol, I've been trying to get out of the habbit of spurting loads of information when not required.

Although a total solution to your requirements doesn't exist, this forum is a good place to get/ build ideas towards finding a solution.  We are all working on various incarnations of AGI's.

As you can see, loads of helpful input already.

If your long term goal was not achievable then we would all be wasting our time and efforts, so ignore what I said and start breaking the problem down.

A semantic tree perhaps like Ranch says, break each task down into it's constituent parts, start at a high resolution then work to finer detail eventually arriving at servo/ machine level instructions. Even building a simple parsed language to hold the order/ instructions that can easily be converted at a later date into any format as your ideas develop. Start getting the nuts and bolts down.

Code: [Select]
Function Pickup_Object (x)
  Look_for (x) - (see below)
  Right_Grasp (x)
  Lift (x)
End Function

Function Look_For (z)
  Load_object_pattern (z)
  Seek_Surface
  Scan_for_pattten (z)
End Function

Edit: Oh! which is just what 8plan.net said... I must read all comments before I spout lol.


I've had the basic notion of decomposition in my head, but just the act of practicing and spelling it out seems to help. One thing that just recently occurred to me is the possibility of using Natural Language Processing to "assemble" the decomposed version (I believe the notion of realizing a concept into concrete code and data models is referred to as reification). In fact, I actually did a very naive toy version of exactly this for a separate project of mine: https://github.com/christabor/MoAL/blob/master/MOAL/software_engineering/reification/reification.py

The idea is to generate a parse tree and then using the different part-of-speech tokens, output actual code from each one. So, in this case, verbs map to functions and nouns map to classes. Unfortunately it's not very robust so the wrong nouns often map to a class, because it does not determine the subject correctly -- but this could be overcome with more robust code.

With this in mind, I could imagine taking all the "tasks" for which I have raw, structured data, analyzing the parts of speech for each discrete task, and then generating pseudocode for all tasks for all jobs. This could be a first stepping stone. The nice thing about this is that duplicate tasks across seemingly disparate jobs could become unified and make it easier to "divide and conquer" the problem space.

I think just talking about the problem is having a positive effect on my thinking!

ranch vermin

  • Not much time left.
  • *******
  • Starship Trooper
  • Posts: 366
  • Not much time.
Re: Non expert development of expert systems?
« Reply #14 on: September 03, 2015, 10:10:16 AM »
Your english wont be complete enough without ALOT of guesswork on behalf of the NLP algorythm, because when we explain things 99% of the activity is omitted from the sentence entirely.

This is an AGI attempt, and its an unsolved problem to date.

Again ill suggest what I suggested before - go to some level of specificy, but dont go any further at first,  itll be alot easier for anyone.
A bit from here, a bit from there, and bring it together and see the whole picture.

 

Welcome

Please login or register.



Login with username, password and session length
The ALTAIR Robot Chronicles
by Tony Ellis (Home Made Robots)
Today at 07:53:36 AM
Thanks Again
by Art (Graphics and Video Software)
April 24, 2017, 01:41:21 AM
AI in cars
by Data (General AI Discussion)
April 24, 2017, 01:24:18 AM
outline from gadient mask
by korrelan (General AI Discussion)
April 23, 2017, 08:16:04 PM
Friday Funny
by Art (General Chat)
April 23, 2017, 07:37:42 PM
Nothing Else Matters
by 8pla.net (General Chat)
April 21, 2017, 05:04:38 AM
Prepare to be Transformed...
by Maviarab (AI in Film and Literature.)
April 20, 2017, 07:18:34 PM
Want to learn artificial intelligence
by yotamarker (AI Programming)
April 20, 2017, 06:48:34 PM
DeepMind
by Art (AI News )
April 19, 2017, 06:19:36 PM
A Murder Posted on Facebook Prompts Outrage and Questions Over Responsibility
by Art (AI News )
April 18, 2017, 10:12:52 AM
AI based 3d scanner
by Art (AI News )
April 18, 2017, 10:04:39 AM
Artificial Earthling
by Art (AI News )
April 14, 2017, 03:49:08 PM
Faster websites with fewer bugs
by Tyler (Robotics News)
April 13, 2017, 10:50:45 PM
New Tools Needed to Track Technology’s Impact on Jobs, Panel Says
by Tyler (AI News )
April 13, 2017, 10:50:43 PM
SMART automation
by Tyler (Robotics News)
April 13, 2017, 04:48:34 PM
Taming data
by Tyler (Robotics News)
April 13, 2017, 10:48:07 AM

Users Online

19 Guests, 0 Users

Most Online Today: 29. Most Online Ever: 208 (August 27, 2008, 08:24:30 AM)

Articles