No internet connection
  1. Home
  2. How to

Mouse click relative to UI element not working with Undertone Plugin

By dave johnson @dave_johnson
    2020-10-16 22:27:00.667Z

    Hello, I'm trying to use the "Mouse click relative to UI element" to click a button on a plugin but it does not appear to be working. I have other plugins that I use this for with no issue. Even on the plugin I'm having issues with the Render and Preview buttons work just fine if I set the X and Y to them.
    The capture button however refuses to click.

    Is this an issue with certain plugins where it doesn't work?

    Thanks for any help.
    Dave

    • 37 replies

    There are 37 replies. Estimated reading time: 19 minutes

    1. Dustin Harris @Dustin_Harris
        2020-10-17 07:13:42.909Z

        Heyya! I’m not at home so I can’t post the code I used, but the undertone plugin UI changes via mouse-moves, so I did mouse set relative, followed by mouse click relative. Hopefully that can point you in a direction until I’m able to post my example. @dario.ramaglia might have some good suggestions too!

        1. In reply todave_johnson:
          Dustin Harris @Dustin_Harris
            2020-10-19 15:19:01.645Z

            OK! Back home now. This is how I've been clicking things, this example clicks the learn button for slot one:

            var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`);
            if (!asWin.exists) {
            
            
                asWin = sf.ui.proTools.audioSuiteOpenPlugin({
                    category: 'Other',
                    name: `Undertone`,
                }).window;
            }
            var screenIndex = 0;
            var screenFrame = sf.ui.screens.allScreens[screenIndex].visibleFrame;
            var windowFrame = asWin.frame;
            
            sf.mouse.setPosition({
                position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 },
            });
            
            sf.mouse.click({
                position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 },
                isCommand: false,
            });
            

            this will 'learn' slot one. If you change isCommand: false, to isCommand: true, it will clear slot one.

            Hopefully this points you in the right direction :)

            1. Ddave johnson @dave_johnson
                2020-10-19 17:17:24.069Z

                This works perfectly!

                Thanks so much.

              • C
                In reply todave_johnson:
                Chris Winter @Chris_Winter
                  2020-10-19 22:16:45.070Z

                  Hi Dustin, as I'm always trying to learn more about this stuff, what do your screenIndex and screenFrame bits of code do?

                  Thanks in advance!

                  1. Dustin Harris @Dustin_Harris
                      2020-10-20 03:46:17.718Z

                      Technically I didn’t need to use screen frame here but I copy/pasted from my larger script that moves the plugin around my screen. ScreenIndex is which physical screen you want to reference in your code. The ‘main’ screen in macOS is always defined as the first screen, so screenIndex 0. Your second screen would be screenIndex 1, etc. screenFrame is the properties of the screen defined in the screenIndex. My screenIndex 0 is a 4K monitor, so I can get its height and width by getting:
                      var screenWidth = screenFrame.w
                      var screenHeight = screenFrame.h
                      Those can be used to position window elements on the screen.

                      Same with asWin.frame, I can use it to get the size and position of my audiosuite window, move it, click elements within the window, etc :)
                      Make sense?

                      1. CChris Winter @Chris_Winter
                          2020-10-20 04:42:51.432Z

                          Yeah it does, thank you.
                          My screen numbers tend to change a bit depending on what's on when the mac boots - I too have a big 4k but it mostly gets used on the post production jobs, and doesn't stay on constantly, so it may not be very reliable for me. I can certainly see the appeal though - thanks for the explanation!

                          1. Dustin Harris @Dustin_Harris
                              2020-10-20 11:56:34.826Z

                              In my case what I’m doing is this: there are several
                              Audiosuite plugins that I use often that take a few seconds to open and configure. So I wrote a script that instead of opening, processing, then closing the plugin, I open, process, then move the plugin to the bottom of the screen, so next time I need it, I can just move the already open plugin to the middle of the screen, process it, then move it back to the bottom of the screen again.

                              1. CChris Winter @Chris_Winter
                                  2020-10-21 02:32:06.646Z

                                  Brilliant! That's almost working out my monitor flipping issues for... :D

                          2. In reply todave_johnson:

                            have you figured out how to put the mouse back to where it started ?

                            1. Dustin Harris @Dustin_Harris
                                2021-04-04 11:55:22.889Z

                                Hadn’t thought of that, but it shouldn’t be difficult if the mouse pointer started on the same screen as the undertone plugin...

                                1. Hey guys.

                                  Just jumping in here to clear some things up.

                                  The coordinate system in macOS is global - across all screens.
                                  So using the sf.mouse.getPosition and setPosition will work to move the mouse and restore it, no matter which screen it originated on and where you're moving it.

                                  Dustin's original code here didn't use the screen frames at all - and neither should it - here I've removed it:

                                  var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`);
                                  if (!asWin.exists) {
                                      asWin = sf.ui.proTools.audioSuiteOpenPlugin({
                                          category: 'Other',
                                          name: `Undertone`,
                                      }).window;
                                  }
                                  var windowFrame = asWin.frame;
                                  
                                  sf.mouse.setPosition({
                                      position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 },
                                  });
                                  
                                  sf.mouse.click({
                                      position: { "x": windowFrame.x + 20, "y": windowFrame.y + windowFrame.h - 46 },
                                      isCommand: false,
                                  });
                                  

                                  What happens in this case is likely that Undertone doesn't react to a quick mouse click, but instead needs you to hover the mouse over the element before it's clicked. That's solved by Dustin's code.
                                  But you shouldn't need to worry about anything related to screens - SF and macOS takes care of all those abstractions for you.

                                  Worrying about screens is only needed if you actually want to move windows to a specific screen. But for clicking relative to a window, you're only dealing in relative coordinates, so no need to think about the global coordinates in those cases.

                                  1. Also note, that if the only issue is calculating the relative position, then mouseClickElement does already have support for calculating it from the bottom left corner:

                                    var asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`);
                                    if (!asWin.exists) {
                                        asWin = sf.ui.proTools.audioSuiteOpenPlugin({
                                            category: 'Other',
                                            name: `Undertone`,
                                        }).window;
                                    }
                                    
                                    asWin.mouseClickElement({
                                        relativePosition: {
                                            x: 20,
                                            y: -46,
                                        },
                                        anchor: 'BottomLeft',
                                    });
                                    
                                    1. In reply tochrscheuer:
                                      Dustin Harris @Dustin_Harris
                                        2021-04-08 12:05:40.292Z

                                        Heyya! Thanks for the explanation! I didn’t realize SoundFlow was dividing the global coordinates into the individual screens in the background, that’s awesome!

                                        So, is that to say when you use the getMousePosition() method (not sure if that is the exact name, I can’t scroll up on my phone) the screen it is on is part of that data?

                                        1. the screen it is on is part of that data

                                          Well.. The pixels are just in one huge coordinate system. The screens don't start at 0,0 - they start at wherever they're positioned in the global space:

                                          In this example, no screens start at 0,0 - 0,0 would be in the top left most area.

                                          1. Note, this is not something SF is doing for you. This is how pixels work on macOS.

                                            1. Dustin Harris @Dustin_Harris
                                                2021-04-08 17:11:06.520Z

                                                If I'm not mistaken, the 0,0 point of the global coordinate system is top left of index[0] which I believe is the screen with the menubar, and coordinates above and left of that point are negative numbers.

                                                1. Gotcha, yea that sounds right

                                      • Dustin Harris @Dustin_Harris
                                          2021-04-04 12:12:36.691Z

                                          Try this:

                                          let originalMousePosition = sf.mouse.getPosition().position;
                                          
                                          //.. do stuff
                                          
                                          sf.mouse.setPosition({
                                              position: { "x": originalMousePosition.x, "y": originalMousePosition.y },
                                          });
                                          
                                          
                                        • In reply todave_johnson:

                                          Thank you, Its on a separate monitor but Ill hack away at it, se if it can work

                                          1. bummers nope, I thnk too many x's and y's !

                                            1. Dustin Harris @Dustin_Harris
                                                2021-04-08 01:48:45.055Z

                                                They key here is finding which screen index you’re starting from programmatically instead of defining it like I have. I haven’t waded in those waters yet but it’s likely a useful exercise for me to undertake. Just need to get over a two-week ADR mountain backlog first :)

                                                1. Dustin Harris @Dustin_Harris
                                                    2021-04-08 18:24:09.737Z

                                                    Another approach is moving the plugin window instead of the mouse. I keep all of my audiosuite plugins at the bottom edge of the screen, and move them into the center of the screen to process, then move them back down

                                                2. In reply todave_johnson:

                                                  On a separate note, we've actually worked a bit with the Undertone developers to get a real integration going, but due to a large workload we didn't get to it at the time.
                                                  I'll see if we can revive this effort :)

                                                    1. BBen Rauscher @Ben_Rauscher
                                                        2024-07-26 14:31:10.063Z

                                                        thread bump. would love to see an official Undertone package.

                                                        1. Dustin Harris @Dustin_Harris
                                                            2024-07-27 12:46:09.806Z

                                                            What sort of things are you looking for?

                                                            1. BBen Rauscher @Ben_Rauscher
                                                                2024-07-27 14:58:52.650Z

                                                                A coded command for Slot 1 capture would be great for dialogue editorial. The mouse click method is spotty, and sometimes does not wait until a proper capture is made.

                                                                Options to set a handle length and clip color after render would be nice bonuses.

                                                                Macro would be... open Undertone, capture profile from selected clip(s), render according to user's handle length, set rendered file's clip color.

                                                                1. Dustin Harris @Dustin_Harris
                                                                    2024-07-27 15:02:38.329Z

                                                                    How long are the sections you’re learning from?
                                                                    Mouse click is the only way to do it so far, but if you’re doing long sections I can probably add a waitForNoModals method…

                                                                    1. BBen Rauscher @Ben_Rauscher
                                                                        2024-07-27 15:05:12.232Z

                                                                        Not long. A pause to verify a successful capture before rendering would be great.

                                                                        1. Dustin Harris @Dustin_Harris
                                                                            2024-07-27 15:11:38.579Z

                                                                            Is ‘not long’ 1 frame, a second, 5 seconds?
                                                                            how long does a typical capture action take on your system?

                                                                            1. BBen Rauscher @Ben_Rauscher
                                                                                2024-07-28 20:56:13.315Z

                                                                                Maybe 1-2 seconds of reference material on average. System is a 2019 Mac mini 32GB ram, so it's still snappy. Sometimes the macro will blow through without pausing for a capture to register, other times it won't. The outcome is essentially a coin toss every time. Would love to have a coded function rather than a GUI mouse click for the Slot 1 capture.

                                                                                1. Dustin Harris @Dustin_Harris
                                                                                    2024-07-28 21:10:37.578Z

                                                                                    I would too, but it's not currently possible with how the plugin is coded, for now it has to be mouse clicks. Post the script you're using here and I'll see if I can improve it for you. Note: handles can have side effects with this plugin, as learning a slot with handles enabled can pull information in from the handles, which isn't always desired when using it to generate fill. But I can add waits to your script to ensure more success with loading and the clip colour change as well...

                                                                                    1. BBen Rauscher @Ben_Rauscher
                                                                                        2024-07-28 21:16:19.874Z

                                                                                        Understood; and thank you for your help.

                                                                                        Second mouse click is to enter 10 second handles to generate extra fill, which has never been an issue for me. Couldn't get any of the Set Value of Text functions to work for that.

                                                                                        1. BBen Rauscher @Ben_Rauscher
                                                                                            2024-07-28 21:16:43.080Z

                                                                                            PS, this is in the Compact view mode.

                                                                                            1. Dustin Harris @Dustin_Harris
                                                                                                2024-07-28 21:19:44.694Z

                                                                                                Ahhh I I think I know your issue now... I'll see if I can find macro actions to achieve the same thing as my script...

                                                                                                1. In reply toBen_Rauscher:
                                                                                                  Dustin Harris @Dustin_Harris
                                                                                                    2024-07-28 21:48:58.977Z

                                                                                                    I wasn't able to find a way to do what I needed with macro actions, but here's a script that (hopefully) does the same as your script but with the necessary adjustments? I can't test it as I don't have the original undertone anymore, but I took this from part of my Undertone 2 script. Let me know if it works!

                                                                                                    /**@typedef {{
                                                                                                     * colorBrightness: "Light" | "Medium" | "Dark",
                                                                                                     * colorNumber: number
                                                                                                     * }} ptColor 
                                                                                                     */
                                                                                                    //end typdef
                                                                                                    
                                                                                                    
                                                                                                    /**@type  ptColor*/
                                                                                                    const fillColor = { colorBrightness: "Medium", colorNumber: 16 }
                                                                                                    
                                                                                                    //string of handle length in seconds
                                                                                                    const HANDLE_LENGTH = "10"
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    function openUnderTonePlugin() {
                                                                                                        try {
                                                                                                            let asWin = sf.ui.proTools.getAudioSuiteWindow(`Undertone`);
                                                                                                    
                                                                                                            if (!asWin.exists) {
                                                                                                    
                                                                                                                asWin = sf.ui.proTools.audioSuiteOpenPlugin({
                                                                                                                    category: 'Other',
                                                                                                                    name: `Undertone`,
                                                                                                                }).window;
                                                                                                    
                                                                                                                asWin.audioSuiteSetOptions({
                                                                                                                    processingInputMode: "EntireSelection",
                                                                                                                    processingOutputMode: "CreateContinuousFile"
                                                                                                                });
                                                                                                    
                                                                                                                if (asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.exists) {
                                                                                                                    var currentHandleLength = asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.value.invalidate().value
                                                                                                    
                                                                                                                    if (currentHandleLength !== HANDLE_LENGTH) {
                                                                                                                        asWin.textFields.whoseTitle.is('Processing Handle Length in Seconds').first.elementClick();
                                                                                                                        sf.keyboard.type({
                                                                                                                            text: HANDLE_LENGTH,
                                                                                                                        });
                                                                                                                        sf.keyboard.press({
                                                                                                                            keys: "enter",
                                                                                                                        });
                                                                                                    
                                                                                                                    }
                                                                                                                }
                                                                                                            }
                                                                                                    
                                                                                                            return asWin
                                                                                                    
                                                                                                        } catch (err) {
                                                                                                    
                                                                                                            throw `error in openUnderTonePlugin(): ${err}`
                                                                                                        }
                                                                                                    }
                                                                                                    
                                                                                                    
                                                                                                    function useSlotOne({ asWin, mode }) {
                                                                                                    
                                                                                                        if (mode !== "learn" && mode !== "clear") {
                                                                                                            log("Invalid status parameter for SlotOne function");
                                                                                                            throw 0;
                                                                                                        }
                                                                                                    
                                                                                                        let command = (mode === "clear");
                                                                                                    
                                                                                                        let windowFrame = asWin.frame;
                                                                                                    
                                                                                                        sf.mouse.setPosition({
                                                                                                            position: { "x": windowFrame.x + 130, "y": windowFrame.y + 300 },
                                                                                                        });
                                                                                                    
                                                                                                        sf.mouse.click({
                                                                                                            position: { "x": windowFrame.x + 130, "y": windowFrame.y + 300 },
                                                                                                            isCommand: command,
                                                                                                        });
                                                                                                    
                                                                                                    }
                                                                                                    
                                                                                                    
                                                                                                    //Open Pro Tools Color Palette
                                                                                                    function openColorPalette() {
                                                                                                        if (sf.ui.proTools.getMenuItem('Window', 'Color Palette').isMenuChecked === false) {
                                                                                                            sf.ui.proTools.menuClick({
                                                                                                                menuPath: ["Window", "Color Palette"],
                                                                                                                targetValue: "Enable",
                                                                                                            });
                                                                                                        }
                                                                                                    
                                                                                                        sf.ui.proTools.windows.whoseTitle.is("Color Palette").first.elementWaitFor();
                                                                                                    
                                                                                                    }
                                                                                                    
                                                                                                    //close Pro Tools Color Palette
                                                                                                    function closeColorPalette() {
                                                                                                    
                                                                                                        sf.ui.proTools.menuClick({
                                                                                                            menuPath: ["Window", "Color Palette"],
                                                                                                            targetValue: "Disable",
                                                                                                        });
                                                                                                    
                                                                                                    }
                                                                                                    
                                                                                                    
                                                                                                    
                                                                                                    /**@param {{ colorNumber: number, colorBrightness: "Light" | "Medium" | "Dark" }} fillColor */
                                                                                                    function doColorChange(fillColor) {
                                                                                                    
                                                                                                        //const mousePosition = sf.mouse.getPosition().position;
                                                                                                    
                                                                                                        let { colorNumber, colorBrightness } = fillColor
                                                                                                    
                                                                                                        openColorPalette();
                                                                                                    
                                                                                                        sf.ui.proTools.invalidate().colorsClick({
                                                                                                            colorTarget: "ClipsInTracks",
                                                                                                            colorType: "Color",
                                                                                                            colorNumber,
                                                                                                            colorBrightness,
                                                                                                    
                                                                                                        });
                                                                                                    
                                                                                                        sf.wait({ intervalMs: 100 });
                                                                                                    
                                                                                                        closeColorPalette();
                                                                                                    }
                                                                                                    
                                                                                                    
                                                                                                    function main() {
                                                                                                    
                                                                                                        
                                                                                                        sf.ui.proTools.appActivateMainWindow();
                                                                                                    
                                                                                                        let underTonePlugin = openUnderTonePlugin();
                                                                                                    
                                                                                                        useSlotOne({ asWin: underTonePlugin, mode: "clear" });
                                                                                                        
                                                                                                        //wait for clear
                                                                                                        sf.wait({ intervalMs: 300 });
                                                                                                        useSlotOne({ asWin: underTonePlugin, mode: "learn" });
                                                                                                        
                                                                                                        //wait 1s for learn to complete
                                                                                                        sf.wait({intervalMs: 1000});
                                                                                                    
                                                                                                        underTonePlugin.audioSuiteRender();
                                                                                                        
                                                                                                        //wait for possible modal progress bar
                                                                                                        sf.wait({intervalMs: 250})
                                                                                                        sf.ui.proTools.waitForNoModals();
                                                                                                        
                                                                                                        underTonePlugin.windowClose();
                                                                                                    
                                                                                                        doColorChange(fillColor);
                                                                                                    }
                                                                                                    
                                                                                                    main();
                                                                                                    
                                                                            2. B
                                                                              In reply todave_johnson:
                                                                              Ben Rauscher @Ben_Rauscher
                                                                                2024-07-30 22:47:01.804Z

                                                                                Thanks! This works great.