MAMEWorld Central >> News
View all threads Index   Threaded Mode Threaded  

Pages: 1

Vas Crabb
BOFH
Reged: 12/13/05
Posts: 3718
Loc: Melbourne, Australia
Send PM


minimaws: a reference -listxml consumer
#368531 - 08/03/17 05:47 PM


There have been some changes to -listxml output, and questions about how to use it from front-end authors. In the interests of ensuring -listxml output is useful, and to provide a reference for front-end developers, I've added a Python-based -listxml consumer called minimaws, in honour of the now-defunct MAME reference web site. It supports some command-line verbs and also provides an interactive web interface. The web interface is probably more interesting for most users and front-end authors.

The minimaws application is in the MAME source tree at scripts/minimaws and should work on any system with MAME (or a copy of its -listxml output), Python 2.7 or Python 3, and a reasonably new version of SQLite (3.6.19 is required for foreign key support). Command-line help is provided for both the main command options and subcommand options (use the -h or --help flag). It's definitely incomplete at this point, but I'm gradually adding functionality to it. I considered a few options for implementing the -listxml consumer, but settled on Python with some of the more interactive features in JavaScript because pretty much everyone has a Python interpreter and a web browser. My apologies if you have trouble reading these languages - they're far from my favourite languages, too. I'm not sure exactly what level of ECMAscript it requires - I've tested it with Firefox 48, and I think the highest feature I use is the 'json' response type for XMLHttpRequest.

Setting up currently requires a couple of steps to make a clean database and load (I intend to streamline this at some point):

Code:

rm minimaws.sqlite3
sqlite3 minimaws.sqlite3
python minimaws.py load path/to/mame



The script uses minimaws.sqlite3 in the working directory as the name for its database unless you override it with the --database option. Loading the database may take a few minutes, but query performance is good. Note that you'll need to provide an actual path with a slash in it if MAME isn't in your PATH (exec treats names with no slashes as commands from somewhere in PATH).

There are a few query commands that work much like MAME's auxiliary verbs, but case sensitive and with better glob behaviour. You should be able to work them out. To run in web server mode, use the serve command:

Code:

python minimaws.py serve



By default it serves HTTP on TCP port 8080 (you can change the port with the --port option). The server is implemented as a WSGI application, so it could also be used in a WSGI container (e.g. Apache mod_wsgi), and it uses cacheable GET requests so it will work behind a caching proxy (e.g. Apache mod_proxy, nginx, or squid). You could theoretically turn most of the pages into static files too, but that would be unwieldy with tens of thousands of files in some directories. While I think I've made it resistant to directory traversal and SQL injection attacks, I open the database in read-only mode if Python 3.4 is available, and I've tried to avoid common sources of vulnerabilities, the code hasn't been audited and may not be completely secure. You probably shouldn't run it directly on a public web server.

If you know the short name of a machine or driver, or you know the path to a source file with device/system definitions, you can jump straight to it with a URL (note that src/mame/drivers is elided in source file paths for historical reasons). You can also get a list of all source files with device/system definitions, although it's a large page and may perform poorly. Pages are interlinked pretty well - you can jump to any device used by a system/device, or any system that uses a device, or parents/clones systems. You can jump to any device defined in a source file, or the source file that defines a device. Example URLs you could start from include:

  • http://localhost:8080/machine/intlc440
  • http://localhost:8080/machine/a2mouse
  • http://localhost:8080/sourcefile/src/devices/cpu/m68000/m68kcpu.cpp
  • http://localhost:8080/sourcefile/


Anyway, the most immediately useful feature for front-end authors, and possibly also for users of computer system emulation, is the ability to show slots options, and update live in response to slot card selection, all straight off the database without having to repeatedly invoke MAME to get updated options. Here's an example of what it looks like, in this case for the fairly simple TI-82:


Although there aren't any in that example, unemulated/imperfect feature flags are shown for devices where applicable. Command-line flags for MAME to produce the selected configuration are shown, including elision of default choices. You can see it in action with any slotted system, for example:

  • http://localhost:8080/machine/ibm5150
  • http://localhost:8080/machine/apple2e
  • http://localhost:8080/machine/ti82


In terms of implementation, the database queries are all in lib/dbaccess.py, and the SAX2 handler that transforms the -listxml output into the normalised database schema is in lib/lxparse.py. The web service is in lib/wsgiserve.py (yes it's a bit ugly because I've forgone a nice templating engine so it will work without any additional modules over a standard Python installation), and static CSS/JavaScript is in lib/assets.

The live slot selection is mostly done in lib/assets/machine.js with a bit of help from the web service. It works as follows:

  • On a page for a system/device with slots, the JavaScript requests slot information for the machine and any devices that can be inserted into slots (fetch_slots). It keeps a map of device slot information (slot_info) and checks whether it has already received or requested details for a device.
  • The only trick in the slot info web service (lib.wsgiserve.SlotsRpcHandler.data_page) is to filter out slots that come from default devices in other slots. This can be done lexically, there's nothing difficult about it.
  • Once all the slot information has been gathered, the slot UI is populated (populate_slots).
  • When a slot card is selected, any subslots are added to the UI (in the callback generated by make_slot_change_handler). This uses the same add_slot_items function that populate_slots uses to create the actual controls and apply defaults.
  • When applying defaults, remember to walk from the topmost level inwards - defaults specified at a higher level take precedence.
  • When a slot card is selected, the JavaScript asynchronously requests emulation status flags if it doesn't have them already (fetch_machine_flags) and adds the information to the table (add_flag_rows).
  • The command-line flag preview is updated in update_cmd_preview after any changes.


All the code is BSD-licensed, so feel free to integrate it into your applications if it'll be useful.



Vas Crabb
BOFH
Reged: 12/13/05
Posts: 3718
Loc: Melbourne, Australia
Send PM


Re: minimaws: a reference -listxml consumer new [Re: Vas Crabb]
#368556 - 08/04/17 09:23 AM


As of d969333, minimaws demonstrates how to expose a little-known MAME feature: slot card BIOS selection. It's possible to select BIOS versions for certain slot devices, mostly Commodore peripherals (note elision of default BIOS on second disk drive, and forcing an empty slot):


Unfortunately this was broken at some point, so it doesn't actually work in MAME 0.188, and the necessary information wasn't being exposed in the -listxml output for devices (as opposed to systems). Anyway, that's all fixed for the next release. If you want to expose this in your frontend and can't follow the logic from the reference implementation, give me a yell and I'll try to walk you through it.

(I've gone to the trouble of eliding defaults in minimaws to make the command line preview look prettier. If you're developing a front-end there's no real need to do the elision step - MAME won't get upset if you supply default options explicitly.)



Vas Crabb
BOFH
Reged: 12/13/05
Posts: 3718
Loc: Melbourne, Australia
Send PM


Re: minimaws: a reference -listxml consumer new [Re: Vas Crabb]
#368558 - 08/04/17 10:28 AM


And now I've improved it so you no longer need the manual schema creation step. Create/load the database with:

Code:

python minimaws.py load --executable path/to/mame




MooglyGuy
Renegade MAME Dev
Reged: 09/01/05
Posts: 1506
Send PM


Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: Vas Crabb]
#368559 - 08/04/17 11:52 AM


> There have been some changes to -listxml output, and questions about how to use it
> from front-end authors. In the interests of ensuring -listxml output is useful, and
> to provide a reference for front-end developers, I've added a Python-based -listxml
> consumer called minimaws, in honour of the now-defunct MAME reference web site. It
> supports some command-line verbs and also provides an interactive web interface. The
> web interface is probably more interesting for most users and front-end authors.
>
> The minimaws application is in the MAME source tree at scripts/minimaws and should
> work on any system with MAME (or a copy of its -listxml output), Python 2.7 or Python
> 3, and a reasonably new version of SQLite (3.6.19 is required for foreign key
> support). Command-line help is provided for both the main command options and
> subcommand options (use the -h or --help flag). It's definitely incomplete at this
> point, but I'm gradually adding functionality to it. I considered a few options for
> implementing the -listxml consumer, but settled on Python with some of the more
> interactive features in JavaScript because pretty much everyone has a Python
> interpreter and a web browser. My apologies if you have trouble reading these
> languages - they're far from my favourite languages, too. I'm not sure exactly what
> level of ECMAscript it requires - I've tested it with Firefox 48, and I think the
> highest feature I use is the 'json' response type for XMLHttpRequest.
>
> Setting up currently requires a couple of steps to make a clean database and load (I
> intend to streamline this at some point):
> rm minimaws.sqlite3
> sqlite3 minimaws.sqlite3
> python minimaws.py load path/to/mame
>
> The script uses minimaws.sqlite3 in the working directory as the name for its
> database unless you override it with the --database option. Loading the database may
> take a few minutes, but query performance is good. Note that you'll need to provide
> an actual path with a slash in it if MAME isn't in your PATH (exec treats names with
> no slashes as commands from somewhere in PATH).
>
> There are a few query commands that work much like MAME's auxiliary verbs, but case
> sensitive and with better glob behaviour. You should be able to work them out. To run
> in web server mode, use the serve command:
> python minimaws.py serve
>
> By default it serves HTTP on TCP port 8080 (you can change the port with the --port
> option). The server is implemented as a WSGI application, so it could also be used in
> a WSGI container (e.g. Apache mod_wsgi), and it uses cacheable GET requests so it
> will work behind a caching proxy (e.g. Apache mod_proxy, nginx, or squid). You could
> theoretically turn most of the pages into static files too, but that would be
> unwieldy with tens of thousands of files in some directories. While I think I've made
> it resistant to directory traversal and SQL injection attacks, I open the database in
> read-only mode if Python 3.4 is available, and I've tried to avoid common sources of
> vulnerabilities, the code hasn't been audited and may not be completely secure. You
> probably shouldn't run it directly on a public web server.
>
> If you know the short name of a machine or driver, or you know the path to a source
> file with device/system definitions, you can jump straight to it with a URL (note
> that src/mame/drivers is elided in source file paths for historical reasons). You can
> also get a list of all source files with device/system definitions, although it's a
> large page and may perform poorly. Pages are interlinked pretty well - you can jump
> to any device used by a system/device, or any system that uses a device, or
> parents/clones systems. You can jump to any device defined in a source file, or the
> source file that defines a device. Example URLs you could start from include:
>
>
> http://localhost:8080/machine/intlc440
> http://localhost:8080/machine/a2mouse
> http://localhost:8080/sourcefile/src/devices/cpu/m68000/m68kcpu.cpp
> http://localhost:8080/sourcefile/
>
>
> Anyway, the most immediately useful feature for front-end authors, and possibly also
> for users of computer system emulation, is the ability to show slots options, and
> update live in response to slot card selection, all straight off the database without
> having to repeatedly invoke MAME to get updated options. Here's an example of what it
> looks like, in this case for the fairly simple TI-82:
>
>
> Although there aren't any in that example, unemulated/imperfect feature flags are
> shown for devices where applicable. Command-line flags for MAME to produce the
> selected configuration are shown, including elision of default choices. You can see
> it in action with any slotted system, for example:
>
>
> http://localhost:8080/machine/ibm5150
> http://localhost:8080/machine/apple2e
> http://localhost:8080/machine/ti82
>
>
> In terms of implementation, the database queries are all in lib/dbaccess.py, and the
> SAX2 handler that transforms the -listxml output into the normalised database schema
> is in lib/lxparse.py. The web service is in lib/wsgiserve.py (yes it's a bit ugly
> because I've forgone a nice templating engine so it will work without any additional
> modules over a standard Python installation), and static CSS/JavaScript is in
> lib/assets.
>
> The live slot selection is mostly done in lib/assets/machine.js with a bit of help
> from the web service. It works as follows:
>
>
> On a page for a system/device with slots, the JavaScript requests slot information
> for the machine and any devices that can be inserted into slots (fetch_slots). It
> keeps a map of device slot information (slot_info) and checks whether it has already
> received or requested details for a device.
> The only trick in the slot info web service (lib.wsgiserve.SlotsRpcHandler.data_page)
> is to filter out slots that come from default devices in other slots. This can be
> done lexically, there's nothing difficult about it.
> Once all the slot information has been gathered, the slot UI is populated
> (populate_slots).
> When a slot card is selected, any subslots are added to the UI (in the callback
> generated by make_slot_change_handler). This uses the same add_slot_items function
> that populate_slots uses to create the actual controls and apply defaults.
> When applying defaults, remember to walk from the topmost level inwards - defaults
> specified at a higher level take precedence.
> When a slot card is selected, the JavaScript asynchronously requests emulation status
> flags if it doesn't have them already (fetch_machine_flags) and adds the information
> to the table (add_flag_rows).
> The command-line flag preview is updated in update_cmd_preview after any changes.
>
>
> All the code is BSD-licensed, so feel free to integrate it into your applications if
> it'll be useful.



StilettoAdministrator
They're always after me Lucky ROMS!
Reged: 03/07/04
Posts: 6167
Send PM


Re: Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: MooglyGuy]
#368563 - 08/04/17 06:25 PM


hah, good, I'm not the only one.

And "elision", ain't that that movie with Matt Damon?



- Stiletto



Vas Crabb
BOFH
Reged: 12/13/05
Posts: 3718
Loc: Melbourne, Australia
Send PM


Re: Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: Stiletto]
#368566 - 08/04/17 06:41 PM


> And "elision", ain't that that movie with Matt Damon?

Fuck, given how many situations where the C++ standard talks about "copy elision" being a valid optimisation, I would've thought C++ developers would be familiar with the word.



MooglyGuy
Renegade MAME Dev
Reged: 09/01/05
Posts: 1506
Send PM


Re: Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: Vas Crabb]
#368569 - 08/05/17 01:07 AM


> > And "elision", ain't that that movie with Matt Damon?
>
> Fuck, given how many situations where the C++ standard talks about "copy elision"
> being a valid optimisation, I would've thought C++ developers would be familiar with
> the word.

Most developers of any language don't actually read the standards, they learn by doing, for better or for worse.



StilettoAdministrator
They're always after me Lucky ROMS!
Reged: 03/07/04
Posts: 6167
Send PM


Re: Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: Vas Crabb]
#368598 - 08/06/17 06:13 AM


> > And "elision", ain't that that movie with Matt Damon?
>
> Fuck, given how many situations where the C++ standard talks about "copy elision"
> being a valid optimisation, I would've thought C++ developers would be familiar with
> the word.

Come on now, I barely count as a C++ developer and you know it.

- Stiletto



BIOS-D
MAME Fan
Reged: 08/07/06
Posts: 1546
Send PM


Re: Bro stop saying "elide", nobody knows what the hell that word means *nt* new [Re: Vas Crabb]
#368604 - 08/06/17 02:53 PM


> Fuck, given how many situations where the C++ standard talks about "copy elision"
> being a valid optimisation, I would've thought C++ developers would be familiar with
> the word.

Optimization must be the last they think after spending 10 hours per day on a routine it didn't work as expected until now. Not mentioning the (always) behind in schedule, the constant redone of work because of bad planning or sudden changes and the acceptable speed for something that won't be a video game, a kernel or an emulator.


Pages: 1

MAMEWorld Central >> News
View all threads Index   Threaded Mode Threaded  

Extra information Permissions
Moderator:  John IV, Tafoid 
1 registered and 35 anonymous users are browsing this forum.
You cannot start new topics
You cannot reply to topics
HTML is enabled
UBBCode is enabled
Thread views: 793