Hey, I'm looking for some help, have done a search first and can't see anything specific to my problem, before we delve in, full disclosure, I am a complete noob to soundflow, and even more of a noob when it comes to coding, so please bear with me.
OK, I'm editing dialogue in protools and once rendered I am uploading into a piece of software that seems like a prototype, I upload the files for each script, tap to dialogue line for the specific character I'm working on, press play to listen to the file for that entry, then I press verify if it is ok and then repeat.
This piece of software then renames the files and spits out a .json file so the audio can be entered into unity.
At the moment the play, verify, stop and any other selectable items are on what SEEM to be buttons.
I created a simple macro that consisted of a key trigger associated to the app, then click UI element. BUT, when I tried to select the ui element it didn't "see" it, the whole screen was highlited pink. I searched the net and found the click ui element video and saw the info relating to clicking command before to access any deeper levels of the app before choosing the UI, this didn't help either, it just kept flagging that no UI element was present
I tried 'Click relative to UI' this didn't work either, it's just a slab of pink for the whole app, like it can't "see" the buttons.
So I went with mouse position, mouse click and this works, but not all of the time, obviously this is very dependant on screen size, so is not ideal, but it did work for half a day with my bastardised hamfisted code, which I "borrowed" from some forum posts. I imagine it's not great reading, but this was to click the PLAY "button":
Like I said, this works, but not always, and I'd prefer a better/more efficient way so as not to have to rely on the screen being the same size or in the same place etc....
Any ideas? I can't really say much more because I've signed an NDA, but the app is VERY basic, but performs it's task perfectly. looks almost like an old web page or an old chat room, I had a look at the contents and it would seem that it follows the unity file structure, and seems to be made up of a few scrollable windows stuck together.
Thanks
Kitch Membery @Kitch2021-02-25 07:33:36.008ZHi Russell,
What you are trying to achieve is difficult to troubleshoot as I would need to get my hands on the application to see if I could look deeper into the UI elements. Does the VO Tools application have a downloadable trial by chance?
Another option would be to see if the application has menu options to trigger "Play" and "Verify", and if so use the "Click Menu Item" Macro Action.
Also, check to see if the application has keyboard shortcuts assigned to the "Play" and "Verify" buttons, and if so you could use keyboard simulation via the "Press Keys" macro action.
Rock on!
- RRussell Miller @Russell_Miller
Hey Kitch, thanks for the reply.
There are no menus and no shortcut keys unfortunately, except for up and down arrow to advance to next line
I've searched the net for the app and it delivers no results, so must be an in-house job. I guess if I stripped the content out, the recorded files etc... I could just send you the app. Where is best to send it to?
Kitch Membery @Kitch2021-02-26 01:57:22.725ZHi Russell,
That's unfortunate that it does not have any menu or keyboard controls. I am not keen on installing software of unknown developers on my system, so let's try and work out another way.
It seems in your code you are pinpointing the coordinates relative to the screen, and not relative to the Application window. You mentioned watching the "Click UI Element" video, but not the "Mouse Click Relative to UI Element" video.
In this video, I walk you through how and when to use the Macro Action "Mouse Click Relative to UI Element".
Maybe this will help you solve the problem.
Let me know how it goes. :-)
Rock on!
- RRussell Miller @Russell_Miller
Hey Kitch, thanks for bearing with me.
Re: Mouse click relative to UI, I did try that a few times before posting, but always worth a refresh. Followed the vid and still not working, BUT, then I changed the click behaviour to "down" instead of "up and down" and this produced the effect of the mouse appearing on the button I want to push and appearing to have the button held down, which is great. As we at least know that the coordinates are correct and the mouse is clicking down. So, there would appear to be a problem with the VOTool app recognising the "up click" or release of the mouse button. For the moment I've duplicated the mouse click relative to UI element action so there are two in my macro, one for down click and another for up click and this seems to be working ok, if a little clunky.So, I wonder why VOTool isn't recognising the up click, or is not reacting fast enough? I will say, that when you hover over buttons in VO tool it takes a moment to recognise them, for them to become highlighted, so logic dictates that that could be an indicator to the problem.
- Kitch Membery @Kitch2021-02-26 19:31:22.532Z
Your logic seems sound to me. It could be how the app was designed.
- In reply toKitch⬆:RRussell Miller @Russell_Miller
Actually, it seems to work fine using 2 seperate relative to ui actions. Is there a way to set the coordinates so they're relative to window size? i.e. So they track with window resizing?
- Kitch Membery @Kitch2021-02-26 19:36:12.041Z
Great to hear you got it working... There is a way to do this. Where are the buttons located in the window... Can you upload a screenshot?
Also If you could paste the script you have so far so I can take a look?
Here is a tutorial for how to quote code in the SoundFlow forum, so that it displays in a readable format:
- RRussell Miller @Russell_Miller
Hey Kitch, is there any way I could send you the screengrab of the programme rather than attach it?
- Kitch Membery @Kitch2021-02-26 19:45:20.287Z
Yes sure, send it to kitch@SoundFlow.org.
- RRussell Miller @Russell_Miller
Here's the code, it's a bastardisation of Jesper Ankarfeldt's EAM code. I thought I'd cracked it, but then resized the window and the mouse positions just stayed static. I played around with the state of x and y being positive or negative, then using a combination of negative and positive numbers along with swapping true for false etc... I did get it to track with a resize for a little while, but extreme resizing exposed errors, i.e. it didn't track.
function clickVOToolinPos(x, y, cornerX, cornerY) { var VOToolinFrame = sf.ui.app('com.ZAUM.wtfuriesVOTool').windows.whoseTitle.is('wtfuriesVOTool').first.frame; var VOToolinX = VOToolinFrame.x + (VOToolinFrame.w * cornerX); var VOToolinY = VOToolinFrame.y + (VOToolinFrame.h * cornerY); sf.ui.app('com.ZAUM.wtfuriesVOTool').windows.whoseTitle.is('wtfuriesVOTool').first.mouseClickElement({ relativePosition: { x: VOToolinX + x, y: VOToolinY + y }, clickType: "Down", clickCount: 1, }); sf.ui.app('com.ZAUM.wtfuriesVOTool').mainWindow.mouseClickElement({ relativePosition: { x: VOToolinX + x, y: VOToolinY + y }, clickType: "Up", }); } clickVOToolinPos(65, 730, false, false);It was working, and was clicking buttons no problem, until I resized my screen, or moved it to another screen, then all bets were off.
- Kitch Membery @Kitch2021-02-26 21:34:52.954Z
Unfortunately, the app is not able to be opened on my system. And so I can't see the way it scales the height and width.
However, this should give you an idea of how to get it working... Obviously this is untested but I don't see why it should not work.;
//I converted the button x position into a percentages by dividing the button x position by the window frames width //I converted the button y position into a percentages by dividing the button y position by the window frames height //The y values may differ depending on how the window resizes. //You will need to update this to add button x & y percentages. const buttonPositions = { play: { x: 0.78, y: 0.29 }, Stop: { x: 0.78, y: 0.29 }, message: { x: 0.78, y: 0.29 }, addAudio: { x: 0.78, y: 0.29 }, sendToFolder: { x: 0.78, y: 0.29 }, notVerified: { x: 0.81, y: 0.38 }, }; function clickVOToolButton(button) { const voTools = sf.ui.app('com.ZAUM.wtfuriesVOTool'); const voToolsWin = voTools.windows.whoseTitle.is('wtfuriesVOTool').first; const voToolsWinFrame = voToolsWin.frame; voTools.appActivateMainWindow(); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y //Change the y value to a fixed value if static. }, clickType: "Down", }); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y //Change the y value to a fixed value if static. }, clickType: "Up", }); } function main() { //Get original mouse position const originaMousePosition = sf.mouse.getPosition().position; //Click button (Change "play" to what ever button you wnat to click window) clickVOToolButton(buttonPositions.play); //Return to mouse position sf.mouse.setPosition({ position: originaMousePosition }); } main();Hope that helps. :-)
- RRussell Miller @Russell_Miller
That works perfectly for the x axis, absoilutely perfect no matter how thin the screen gets. Amazing!
The Y axis doesn't hold its position when resized though, so I'm guessing this is because the script is taking it's value from the opposite end that the app calculates its resize from? so, at the moment I'm calculating from the top of the app down, and that is the way the script is calculating too, but I'm guessing that the window resizes from the bottom up? That might explain why I had some luck with changing +- in the original script maybe?- Kitch Membery @Kitch2021-02-26 22:53:54.303Z
Did you mean "The Y axis doesn't change its position when resized"?
If so, try this;
//You will need to update this to add button x percentage & y position. const buttonPositions = { play: { x: 0.78, y: 300 }, Stop: { x: 0.78, y: 300 }, message: { x: 0.78, y: 300 }, addAudio: { x: 0.78, y: 300 }, sendToFolder: { x: 0.78, y: 300 }, notVerified: { x: 0.81, y: 380 }, }; function clickVOToolButton(button) { const voTools = sf.ui.app('com.ZAUM.wtfuriesVOTool'); const voToolsWin = voTools.windows.whoseTitle.is('wtfuriesVOTool').first; const voToolsWinFrame = voToolsWin.frame; voTools.appActivateMainWindow(); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: button.y }, clickType: "Down", }); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: button.y }, clickType: "Up", }); } function main() { //Get original mouse position const originaMousePosition = sf.mouse.getPosition().position; //Click button (Change "play" to what ever button you wnat to click window) clickVOToolButton(buttonPositions.play); //Return to mouse position sf.mouse.setPosition({ position: originaMousePosition }); } main();- RRussell Miller @Russell_Miller
Oh, er, no, the mouse position on the x axis is great regardless of screen width, the percentages are working for x so that seems fine, but mouse position on y axis moves above or below button position on y axis when the screen is resized, which makes me think , could this be because the relative position based on percentage is being calculated from the top of the app screen but the app is resizing from the bottom? so instead of .26 taken from the top I actually need to be measuring from the bottom and using .76, but telling the script to measure from the bottom? Does that make sense?
- RRussell Miller @Russell_Miller
Or does it not matter? I should probably go to bed, simple maths is becoming a challenge.
- In reply toRussell_Miller⬆:Kitch Membery @Kitch2021-02-26 23:11:08.946Z
Sorry Russell, I'm a bit confused. Have you tried switching the y axis percentage in the
buttonPositionsobject at the beginning of the script?- RRussell Miller @Russell_Miller
Hey Kitch, sorry I was super tired and probably not making much sense.
I've got it working though, I I figured the title bar never resizes, and if that very top corner is where the app is taking it's window frame measurement from then this needs to be added to make the percentage for each 'Button' work no matter the screen resize, so I simply measured that (25) using shift command 4, then remeasured each button using same method, but taking my measurement upto the title bar, NOT including it and then placed that into the code like so:const buttonPositions = { play: { x: 0.777, y: 0.159 }, Stop: { x: 0.78, y: 0.185 }, message: { x: 0.78, y: 0.185 }, addAudio: { x: 0.78, y: 0.185 }, sendToFolder: { x: 0.78, y: 0.185 }, verified: { x: 0.777, y: 0.38 }, }; function clickVOToolButton(button) { const voTools = sf.ui.app('com.ZAUM.wtfuriesVOTool'); const voToolsWin = voTools.windows.whoseTitle.is('wtfuriesVOTool').first; const voToolsWinFrame = voToolsWin.frame; voTools.appActivateMainWindow(); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y +24 }, clickType: "Down", }); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y +24 }, clickType: "Up", }); } function main() { const originaMousePosition = sf.mouse.getPosition().position; clickVOToolButton(buttonPositions.play); sf.mouse.setPosition({ position: originaMousePosition }); } main();It now works perfectly regardless of screen size or even which screen I drag the app to.
Thankyou so much Kitch, you have been so helpful and generous with your time and expertise. I have done zero coding before, and I guess I've done zero now as it was all you and initially borrowed code from Anders.
Thanks again.
- RRussell Miller @Russell_Miller
OK, so, the plot thickens, but I've truly sorted it now.
In the script attached to my last reply the positions were working great and tracked with all screen resizes, which is brilliant. But then I realised ift wasn't actually registering clicks, it's like it needs time to 'see' the mouse pointer for some reason. I'd noticed that once a button had been selected pressing 'space' activated that button, so I placed a script to press space inbetween the down click and upclick portions of the script. It now works perfectly and I've attached in case anyone encounters an app with similar idiosyncracies.play: { x: 0.777, y: 0.159 }, Stop: { x: 0.78, y: 0.185 }, message: { x: 0.78, y: 0.185 }, addAudio: { x: 0.78, y: 0.185 }, sendToFolder: { x: 0.78, y: 0.185 }, verified: { x: 0.777, y: 0.38 }, }; function clickVOToolButton(button) { const voTools = sf.ui.app('com.ZAUM.wtfuriesVOTool'); const voToolsWin = voTools.windows.whoseTitle.is('wtfuriesVOTool').first; const voToolsWinFrame = voToolsWin.frame; voTools.appActivateMainWindow(); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y +24 }, clickType: "Down", }); sf.keyboard.press({ keys: "space",}); voToolsWin.mouseClickElement({ relativePosition: { x: voToolsWinFrame.w * button.x, y: voToolsWinFrame.h * button.y +24 }, clickType: "Up", }); } function main() { const originaMousePosition = sf.mouse.getPosition().position; clickVOToolButton(buttonPositions.play); sf.mouse.setPosition({ position: originaMousePosition }); } main();Thanks again Kitch for all your help, tmy mind is now racing with the possibilities for controlling other pieces of software, perhaps even creating a decent plugin controller that actually feels like having your hands on a real bit of gear.
- Kitch Membery @Kitch2021-02-27 21:12:26.610Z
You are totally welcome @Russell_Miller. So glad you were able to get it working. And yes that is very strange behaviour that the VO app is displaying.
I'm looking forward to seeing what you come up with.
Rock on!