Mouse click relative to UI element not working with Undertone Plugin
Hello, I'm trying to use the "Mouse click relative to UI element" to click a button on a plugin but it does not appear to be working. I have other plugins that I use this for with no issue. Even on the plugin I'm having issues with the Render and Preview buttons work just fine if I set the X and Y to them.
The capture button however refuses to click.
Is this an issue with certain plugins where it doesn't work?
Thanks for any help.
Dave
- Dustin Harris @Dustin_Harris
Heyya! I’m not at home so I can’t post the code I used, but the undertone plugin UI changes via mouse-moves, so I did mouse set relative, followed by mouse click relative. Hopefully that can point you in a direction until I’m able to post my example. @dario.ramaglia might have some good suggestions too!
- In reply todave_johnson⬆:Dustin Harris @Dustin_Harris
OK! Back home now. This is how I've been clicking things, this example clicks the learn button for slot one:
var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`); if (!asWin.exists) { asWin = sf.ui.proTools.audioSuiteOpenPlugin({ category: 'Other', name: `Undertone`, }).window; } var screenIndex = 0; var screenFrame = sf.ui.screens.allScreens[screenIndex].visibleFrame; var windowFrame = asWin.frame; sf.mouse.setPosition({ position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 }, }); sf.mouse.click({ position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 }, isCommand: false, });
this will 'learn' slot one. If you change
isCommand: false,
toisCommand: true,
it will clear slot one.Hopefully this points you in the right direction :)
- Ddave johnson @dave_johnson
This works perfectly!
Thanks so much.
- CIn reply todave_johnson⬆:Chris Winter @Chris_Winter
Hi Dustin, as I'm always trying to learn more about this stuff, what do your screenIndex and screenFrame bits of code do?
Thanks in advance!
Dustin Harris @Dustin_Harris
Technically I didn’t need to use screen frame here but I copy/pasted from my larger script that moves the plugin around my screen. ScreenIndex is which physical screen you want to reference in your code. The ‘main’ screen in macOS is always defined as the first screen, so screenIndex 0. Your second screen would be screenIndex 1, etc. screenFrame is the properties of the screen defined in the screenIndex. My screenIndex 0 is a 4K monitor, so I can get its height and width by getting:
var screenWidth = screenFrame.w
var screenHeight = screenFrame.h
Those can be used to position window elements on the screen.Same with asWin.frame, I can use it to get the size and position of my audiosuite window, move it, click elements within the window, etc :)
Make sense?- CChris Winter @Chris_Winter
Yeah it does, thank you.
My screen numbers tend to change a bit depending on what's on when the mac boots - I too have a big 4k but it mostly gets used on the post production jobs, and doesn't stay on constantly, so it may not be very reliable for me. I can certainly see the appeal though - thanks for the explanation!Dustin Harris @Dustin_Harris
In my case what I’m doing is this: there are several
Audiosuite plugins that I use often that take a few seconds to open and configure. So I wrote a script that instead of opening, processing, then closing the plugin, I open, process, then move the plugin to the bottom of the screen, so next time I need it, I can just move the already open plugin to the middle of the screen, process it, then move it back to the bottom of the screen again.- CChris Winter @Chris_Winter
Brilliant! That's almost working out my monitor flipping issues for... :D
- In reply todave_johnson⬆:Christopher Barnett @Christopher_Barnett
have you figured out how to put the mouse back to where it started ?
Dustin Harris @Dustin_Harris
Hadn’t thought of that, but it shouldn’t be difficult if the mouse pointer started on the same screen as the undertone plugin...
Christian Scheuer @chrscheuer2021-04-08 10:40:11.541Z
Hey guys.
Just jumping in here to clear some things up.
The coordinate system in macOS is global - across all screens.
So using the sf.mouse.getPosition and setPosition will work to move the mouse and restore it, no matter which screen it originated on and where you're moving it.Dustin's original code here didn't use the screen frames at all - and neither should it - here I've removed it:
var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`); if (!asWin.exists) { asWin = sf.ui.proTools.audioSuiteOpenPlugin({ category: 'Other', name: `Undertone`, }).window; } var windowFrame = asWin.frame; sf.mouse.setPosition({ position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 }, }); sf.mouse.click({ position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 }, isCommand: false, });
What happens in this case is likely that Undertone doesn't react to a quick mouse click, but instead needs you to hover the mouse over the element before it's clicked. That's solved by Dustin's code.
But you shouldn't need to worry about anything related to screens - SF and macOS takes care of all those abstractions for you.Worrying about screens is only needed if you actually want to move windows to a specific screen. But for clicking relative to a window, you're only dealing in relative coordinates, so no need to think about the global coordinates in those cases.
Christian Scheuer @chrscheuer2021-04-08 10:59:50.198Z
Also note, that if the only issue is calculating the relative position, then mouseClickElement does already have support for calculating it from the bottom left corner:
var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`); if (!asWin.exists) { asWin = sf.ui.proTools.audioSuiteOpenPlugin({ category: 'Other', name: `Undertone`, }).window; } asWin.mouseClickElement({ relativePosition: { x: 20, y: -46, }, anchor: 'BottomLeft', });
- In reply tochrscheuer⬆:
Dustin Harris @Dustin_Harris
Heyya! Thanks for the explanation! I didn’t realize SoundFlow was dividing the global coordinates into the individual screens in the background, that’s awesome!
So, is that to say when you use the getMousePosition() method (not sure if that is the exact name, I can’t scroll up on my phone) the screen it is on is part of that data?
Christian Scheuer @chrscheuer2021-04-08 14:55:50.337Z
the screen it is on is part of that data
Well.. The pixels are just in one huge coordinate system. The screens don't start at 0,0 - they start at wherever they're positioned in the global space:
In this example, no screens start at 0,0 - 0,0 would be in the top left most area.
Christian Scheuer @chrscheuer2021-04-08 14:56:20.687Z
Note, this is not something SF is doing for you. This is how pixels work on macOS.
Dustin Harris @Dustin_Harris
If I'm not mistaken, the 0,0 point of the global coordinate system is top left of index[0] which I believe is the screen with the menubar, and coordinates above and left of that point are negative numbers.
Christian Scheuer @chrscheuer2021-04-08 18:05:19.289Z
Gotcha, yea that sounds right
- In reply toChristopher_Barnett⬆:
Dustin Harris @Dustin_Harris
Try this:
let originalMousePosition = sf.mouse.getPosition().position; //.. do stuff sf.mouse.setPosition({ position: { "x": originalMousePosition.x, "y": originalMousePosition.y }, });
- In reply todave_johnson⬆:Christopher Barnett @Christopher_Barnett
Thank you, Its on a separate monitor but Ill hack away at it, se if it can work
Christopher Barnett @Christopher_Barnett
bummers nope, I thnk too many x's and y's !
Dustin Harris @Dustin_Harris
They key here is finding which screen index you’re starting from programmatically instead of defining it like I have. I haven’t waded in those waters yet but it’s likely a useful exercise for me to undertake. Just need to get over a two-week ADR mountain backlog first :)
- In reply toChristopher_Barnett⬆:
Dustin Harris @Dustin_Harris
Another approach is moving the plugin window instead of the mouse. I keep all of my audiosuite plugins at the bottom edge of the screen, and move them into the center of the screen to process, then move them back down
- In reply todave_johnson⬆:Christian Scheuer @chrscheuer2021-04-08 10:49:09.645Z
On a separate note, we've actually worked a bit with the Undertone developers to get a real integration going, but due to a large workload we didn't get to it at the time.
I'll see if we can revive this effort :)Christian Scheuer @chrscheuer2021-04-08 10:49:36.890Z
cc @dieffe
- BBen Rauscher @Ben_Rauscher
thread bump. would love to see an official Undertone package.
Dustin Harris @Dustin_Harris
What sort of things are you looking for?
- BBen Rauscher @Ben_Rauscher
A coded command for Slot 1 capture would be great for dialogue editorial. The mouse click method is spotty, and sometimes does not wait until a proper capture is made.
Options to set a handle length and clip color after render would be nice bonuses.
Macro would be... open Undertone, capture profile from selected clip(s), render according to user's handle length, set rendered file's clip color.
Dustin Harris @Dustin_Harris
How long are the sections you’re learning from?
Mouse click is the only way to do it so far, but if you’re doing long sections I can probably add a waitForNoModals method…- BBen Rauscher @Ben_Rauscher
Not long. A pause to verify a successful capture before rendering would be great.
Dustin Harris @Dustin_Harris
Is ‘not long’ 1 frame, a second, 5 seconds?
how long does a typical capture action take on your system?- BBen Rauscher @Ben_Rauscher
Maybe 1-2 seconds of reference material on average. System is a 2019 Mac mini 32GB ram, so it's still snappy. Sometimes the macro will blow through without pausing for a capture to register, other times it won't. The outcome is essentially a coin toss every time. Would love to have a coded function rather than a GUI mouse click for the Slot 1 capture.
Dustin Harris @Dustin_Harris
I would too, but it's not currently possible with how the plugin is coded, for now it has to be mouse clicks. Post the script you're using here and I'll see if I can improve it for you. Note: handles can have side effects with this plugin, as learning a slot with handles enabled can pull information in from the handles, which isn't always desired when using it to generate fill. But I can add waits to your script to ensure more success with loading and the clip colour change as well...
- BBen Rauscher @Ben_Rauscher
Understood; and thank you for your help.
Second mouse click is to enter 10 second handles to generate extra fill, which has never been an issue for me. Couldn't get any of the Set Value of Text functions to work for that.
- BBen Rauscher @Ben_Rauscher
PS, this is in the Compact view mode.
Dustin Harris @Dustin_Harris
Ahhh I I think I know your issue now... I'll see if I can find macro actions to achieve the same thing as my script...
- In reply toBen_Rauscher⬆:
Dustin Harris @Dustin_Harris
I wasn't able to find a way to do what I needed with macro actions, but here's a script that (hopefully) does the same as your script but with the necessary adjustments? I can't test it as I don't have the original undertone anymore, but I took this from part of my Undertone 2 script. Let me know if it works!
/**@typedef {{ * colorBrightness: "Light" | "Medium" | "Dark", * colorNumber: number * }} ptColor */ //end typdef /**@type ptColor*/ const fillColor = { colorBrightness: "Medium", colorNumber: 16 } //string of handle length in seconds const HANDLE_LENGTH = "10" function openUnderTonePlugin() { try { let asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`); if (!asWin.exists) { asWin = sf.ui.proTools.audioSuiteOpenPlugin({ category: 'Other', name: `Undertone`, }).window; asWin.audioSuiteSetOptions({ processingInputMode: "EntireSelection", processingOutputMode: "CreateContinuousFile" }); if (asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.exists) { var currentHandleLength = asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.value.invalidate().value if (currentHandleLength !== HANDLE_LENGTH) { asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.elementClick(); sf.keyboard.type({ text: HANDLE_LENGTH, }); sf.keyboard.press({ keys: "enter", }); } } } return asWin } catch (err) { throw `error in openUnderTonePlugin(): ${err}` } } function useSlotOne({ asWin, mode }) { if (mode !== "learn" && mode !== "clear") { log("Invalid status parameter for SlotOne function"); throw 0; } let command = (mode === "clear"); let windowFrame = asWin.frame; sf.mouse.setPosition({ position: { "x": windowFrame.x + 130, "y": windowFrame.y + 300 }, }); sf.mouse.click({ position: { "x": windowFrame.x + 130, "y": windowFrame.y + 300 }, isCommand: command, }); } //Open Pro Tools Color Palette function openColorPalette() { if (sf.ui.proTools.getMenuItem('Window', 'Color Palette').isMenuChecked === false) { sf.ui.proTools.menuClick({ menuPath: ["Window", "Color Palette"], targetValue: "Enable", }); } sf.ui.proTools.windows.whoseTitle.is("Color Palette").first.elementWaitFor(); } //close Pro Tools Color Palette function closeColorPalette() { sf.ui.proTools.menuClick({ menuPath: ["Window", "Color Palette"], targetValue: "Disable", }); } /**@param {{ colorNumber: number, colorBrightness: "Light" | "Medium" | "Dark" }} fillColor */ function doColorChange(fillColor) { //const mousePosition = sf.mouse.getPosition().position; let { colorNumber, colorBrightness } = fillColor openColorPalette(); sf.ui.proTools.invalidate().colorsClick({ colorTarget: "ClipsInTracks", colorType: "Color", colorNumber, colorBrightness, }); sf.wait({ intervalMs: 100 }); closeColorPalette(); } function main() { sf.ui.proTools.appActivateMainWindow(); let underTonePlugin = openUnderTonePlugin(); useSlotOne({ asWin: underTonePlugin, mode: "clear" }); //wait for clear sf.wait({ intervalMs: 300 }); useSlotOne({ asWin: underTonePlugin, mode: "learn" }); //wait 1s for learn to complete sf.wait({intervalMs: 1000}); underTonePlugin.audioSuiteRender(); //wait for possible modal progress bar sf.wait({intervalMs: 250}) sf.ui.proTools.waitForNoModals(); underTonePlugin.windowClose(); doColorChange(fillColor); } main();
- BIn reply todave_johnson⬆:Ben Rauscher @Ben_Rauscher
Thanks! This works great.