UI Elements not visible by SF, but are through other methods - (hWnd possible?)
Heya, I'm trying to understand something here.
I've found that UI elements in UAD Luna are very limited to SF (Limited to nothing but the menu paths actually) - I've tried using Xcode Accessibilty Browser and UI Browser to see what's actually being presented by UA. But both seem to reveal (or confirm rather), nothing but the menu path.
I have however noticed recently on a remote session via Zoom, that within Zooms Screen Share selection, I not only had Lunas main Windopw as an option. But also every plugin window too. A little digging and I (think) I've found that Zoom uses Window Handles (hWnd) to source the elements availble to share. Is this something that I might be able to leverage in SF to gain some form control and relative positioning etc? Or if there's a route, maybe via applescipt or terminal to bring some data into SF?
Thanks.
Adam.
Christian Scheuer @chrscheuer2021-10-20 14:11:00.724ZHi Adam,
That's an interesting thought. hWnd is a Windows-specific technology, so it's not what UAD uses on the Mac. But of course it's possible they're using some other technology that could still be helpful.
This being said, SoundFlow can recognize all open windows of an app, always – it's the content of the windows that is only available if the app implements Accessibility.
- AAdz Beath @Adz_Beath
"can recognise all open windows of an app, always"...
Maybe I've hit a bug? - The UI Picker refuses to 'see' anything from Luna except the Menu Paths. The red highlighting doesn't react to any windows relating to Luna (most other apps seem fine). And a click anywhere on an unrecognised window returns an error "Could not pick UI element: Could not find UI element at the specified position".
Christian Scheuer @chrscheuer2021-10-21 11:15:52.155ZHm, can you try clicking the title bar of the window to see if that helps?
- AAdz Beath @Adz_Beath
Not in there now but I know I’ve tried clicking the title bar, edges, and more! It’s kinda like it doesn’t exist!
- In reply tochrscheuer⬆:AAdz Beath @Adz_Beath
I've popped a little vid here of her behaviour @chrscheuer. So none of the Pickers seem to work across many macros. I've done more extensive, tedious hunting for a 'click point'. Nothing except the menu bar is visible even via the Accs Inspector (1 Child) - However Zoom recognises all windows. Main, and Plugin... But ONLY when the Luna is not in focus... ???? Anyway, this ain't about zoom, I did want to illustrate that the elements are visible somehow. I'm just super curious as to your comment that "SoundFlow can recognize all open windows of an app, always". If that's the case then do I have a bug?
Cheers.
Adam.- Christian Scheuer @chrscheuer2021-10-26 14:05:23.917Z
Ah I see. Thanks for sharing this. Yea this issue occurs because Luna is using completely custom drawn windows with no titlebar from macOS.
- AAdz Beath @Adz_Beath
Thanks Christian, any ideas how zoom might be getting a hold of them? Really keen to try and get some control of this in SF. It’s the only barrier I’m facing to a bunch of packages I’d love to start building! 🥺
The pixel scraping via canvas was a thought for a workaround I brought up in another thread but it proved a bit too much in the “around” department! Not very elegant.
Thoughts on where I could chase this from here? Hoping it’s not a pipe dream 🤷♂️- Christian Scheuer @chrscheuer2021-10-26 23:25:51.864Z
Try to run this script while you have Luna focused to see if SF recognizes its windows:
log('Bundle ID is: ' + sf.ui.frontmostApp.activeBundleID); for(let window of sf.ui.frontmostApp.windows.slice()) log('Open window: ' + window.title.value);It should log the bundle ID of Luna and the titles of all open windows.
- AAdz Beath @Adz_Beath
Cheers Christian, Just tried this. Assigning the script to F13 I get the following.
In the SF app log:
And Console logs:
27.10.2021 13:04:58.89 <info> [Backend]: Command accepted for app: com.uaudio.luna 27.10.2021 13:04:58.89 <info> [Backend]: #Key: f13 (105) -> Luna Bundle ID [ckv8vfhrl00013710frbiy678] Validating command... user:cktkz2z5z0000hw10v4ltlq9f:ckv8vfb9w00003710kus97qer Invoking command...user:cktkz2z5z0000hw10v4ltlq9f:ckv8vfb9w00003710kus97qer >> Command: Luna Bundle ID [user:cktkz2z5z0000hw10v4ltlq9f:ckv8vfb9w00003710kus97qer] 27.10.2021 13:04:58.89 <info> [Backend]: [LOG] Bundle ID is: com.uaudio.luna << Command: Luna Bundle ID [user:cktkz2z5z0000hw10v4ltlq9f:ckv8vfb9w00003710kus97qer] 27.10.2021 13:04:59.06 <info> [Backend]: Command accepted for app: com.uaudio.luna
This is with Luna's Main window open and in focus, and 3 plugin windows open (same plugs/test project as in the video).
- Christian Scheuer @chrscheuer2021-10-27 10:45:29.964Z
Gotcha, yea this really does indicate SF (and thus, macOS accessibility API) cannot see the windows at all.
- AAdz Beath @Adz_Beath
Yeah, thanks for the help and digging assist @chrscheuer :-)
I'm going to reach out to UA and see if they can offer any insight. I haven't done already as I'd like to know a little more before asking with this kind of product questioning. It's not really something in the Customer Service department, QA field that I'm confident to get an answer on. More of a developer question that they don't really offer a direct email line to!!!
Had I not seen that Zoom is capable of seeing and tracking the elements I probably would have left it. But I can't help but wonder, it's clearly possible... Somehow!
I'll post back with any results I further uncover.
Thanks again for the help.Adam
- Christian Scheuer @chrscheuer2021-10-28 02:11:34.150Z
Awesome! Thanks for helping push this, and please keep me posted :)
- AAdz Beath @Adz_Beath
Hiya @chrscheuer,
Just an update on the digging at this end.I reached out to UA, refering all the info I had gathered and links to here, and also a breif discussion with the team at nOb. Their nAc software was interesting to me as it seems to use a similar UI method of storing and recalling presets for mouse positioning/window recal etc to then use their nOb Controller (HID) to execute your movements and so on - https://www.nobcontrol.com/nac
Initially, (when I reached out to them (nOb)) they couldn't confirm Luna's full functionality with the nAc software - But Michael from UA CS chased up the conversation, and after some testing between the two companies, they found the same results. nAc was unable to properly recall windows (for exactly the same reasons as SF. No access to UI elements). They do believe that it's a simple unlocking of an (accessibilty) feature with Luna's current UI Framework. So hopefully it's something that might be implemented in the future as it may be a simple change on the developement teams behalf. Fingers crossed on that one for now.
Luna has a great Feedback form within the app that has served as a sort of polling booth for what the developers prioritise on their roadmap. The have done an incredible job at sticking to this, bringing the most requested functions to users based on popularity. I'm submitting this UI element feature frequently but I do fear that myself (and the small number of SF/Luna users out there) are a rather small drop in the ocean of requests!! I've dropped a mention of SF in some of the Luna communities out there. Hopefully some more people might jump on board and get a ball rolling.
I'm hoping to get a package or two, and some video guides together that are as "plug and play" as possible (to avoid putting people off - coding can be scary! Even with macros to some!!) - Hopefully then some others may start to use their Trials and see the benifits on offer - Bringing them into the fray!
I have a busy few weeks ahead of me, but coming into the new year I think it's safe to say you'll be seeing a few more "help me/how do I?" posts from me as I try and get some of these packages a little more user friendly and universal outside of my own rig!!
The Zoom aspect still plagues my head on a daily basis - UA havn't really addressed this when I've asked and I actually don't expect them get too far with trying to reach out to Zoom Development directly (they're not as likely to be as comminactive as nOb I'd guess?).
I have in the meantime asked UA if they can specify what UI framewrok they are using (though this might not be something they'd be willing to disclose to a mere punter :-x ). Maybe with this knowledge there might be another way to find gain access, as Zoom is (this kinda goes back to my original query about hWnd). Still awaiting an answer regarding this.
Anyway, ramble over. Thanks again.... As you were. :-)
Adam.
- Christian Scheuer @chrscheuer2021-11-08 10:37:59.337Z
Wonderful research and digging! Thank you so much for all this.
If you have a direct contact at UA, please let them know I'd be happy to talk to their team directly in case they want to explore this.- AAdz Beath @Adz_Beath
I don't have a direct contact with UA, just a developed thread about this with Michael from CS. I have seen some of the developers on the UA forums so I might try and reach out directly there. I recall @Chris_Wilson posting regarding a contact when I first started using SF.
UAD Luna - Soundflow - Is there a future for this relationship?! #post-8
Was this point of contact ever touched on?Adam.
- Christian Scheuer @chrscheuer2021-11-08 11:03:28.916Z
Thanks for the reminder! I can see in an old thread I do have an email for somebody on the UA team who were interested in us reaching out, so I'll do that.
- AAdz Beath @Adz_Beath
Awesome! - Let me know if there's anything I do to help, test or provide more info with.
- In reply tochrscheuer⬆:AAdz Beath @Adz_Beath
Just heard back from Michael, no big secret thankfully. - Luna uses the Juce UI Framework….
My shovel is in hand 😅
- Christian Scheuer @chrscheuer2021-11-08 22:03:45.269Z
That's great. From Juce 6.1, accessibility is supported by default. So if they make sure to build with 6.1 and aren't actively turning off accessibility, SF should gain full access.
- AAdz Beath @Adz_Beath
That's good news indeed! Let me know if there's any insight to this from the "contact" ;-)
Adam.